Visualising portfolio risk and performance reviews - presentation delivered at the AIPM 2017 National Conference

Written by Bryan Fenech, Director - PPM Intelligence.

Note: This article is a summary of a presentation that I gave at the recent AIPM 2017 Conference in Melbourne. A link to the full presentation is provided at the end.

Traditional approaches to portfolio risk and performance review

More often than not the implementations of portfolio management that I come across do not deliver on their promised benefits. The reason for this is almost always the same: portfolio governance boards and portfolio management offices (PMOs) do not make sufficient use of the plethora of data available to them to support decision making about portfolio selection, monitoring and controlling.

Organisations are particularly weak in the areas of portfolio performance, risk and benefits management. The most common approach followed when portfolio governance boards review their inflight portfolio is to run through a list of projects and programs with health ratings for a set of simple performance metrics along with some basic commentary. The health ratings are usually represented as traffic lights – red, amber, green. The performance metrics usually include overall, budget, schedule, resources, risk, benefits and overall health.If you are doing this you are wasting your time!

Firstly, it is duplication of the governance practices that occur at project and program Steering Committee level and adds little if any value. If something is going wrong with one of the projects or programs in the list the best that the portfolio governance board can do is authorise actions to be taken with respect to the project or program concerned. The portfolio governance board is doing little more than delegating back (with interest) what the Sponsor has escalated up to them and is already doing their best to deal with.

Secondly, and worse still, it is ineffective as it does not make it easy to identify and resolve systemic issues that impact across the portfolio, for eg resource bottlenecks in certain skill categories causing delays in multiple projects, or common risk types and estimation errors. This misses the opportunity for creating learning and facilitating evidence-based decision making that improves structural and systemic problems and inefficiencies regarding the organisation’s portfolio management capability and capacity. This situation is not a good use of anyone’s time and energy.

Portfolio data analytics

An alternative approach is to give your portfolio data to a data analyst with project management knowledge and ask them to turn it into valuable information that tells you something about portfolio performance, risk profile and benefits position as a whole, rather than project by project.

Here are a few examples of what you might ask them to provide you:

1. What types of risk are the most common across the portfolio? Is there a particular type of resource bottleneck that presents regularly? Is vendor management a recurring problem? Is there evidence that project managers are consistently demonstrating forms of cognitive bias in their estimation, such as being overly optimistic?
2. Are there particular “break points” in the portfolio (ie, projects or programs upon which there is a high degree of dependency by other projects and programs)? What is the health of the most significant break points? What is the combined value of 'at risk' investment dollars or benefits dependent on these break points?
3. How does the health of tier 1 projects and programs compare with less critical projects and programs? Is there a point of increasing complexity at which portfolio performance and throughput tends to 'fall off a cliff'? Is the organisation allowing sufficient schedule and budget contingency in its tier 1 projects for these realities?
4. What is the combined value of investment dollars or benefits by overall traffic light rating? Is the position improving over time?

The information derived can be a snapshot at a point in time or represent historical trends.

Portfolio data visualisation

However, it is critical that portfolio analytics are presented in a way that decision makers can make immediate sense of them. Data visualisation techniques can help to present an incredibly rich picture of your project portfolio. Bringing together graphical representations of portfolio information in a portfolio info-graphic facilitates immediate practical application by senior decision makers. This approach delivers vastly superior outcomes in terms of the quality of learning and decision making.

Charles Minard’s famous graphical depiction of Napoleon’s campaign against Russia helps us to think about how we might represent portfolio information graphically. The Minard diagram shows the losses suffered by Napoleon's army in the 1812–1813 period. Six variables are plotted: the size of the army, its location on a two-dimensional surface (x and y), time, direction of movement, and temperature. The line width illustrates a comparison (size of the army at points in time) while the temperature axis suggests a cause of the change in army size. This multivariate display on a two dimensional surface tells a story that can be grasped immediately while identifying the source data to build credibility.

Types of visualisation

Author Stephen Few described eight types of quantitative messages that users may attempt to understand or communicate from a set of data and the associated graphs used to help communicate the message. I’ve adapted his typology so that it applies to portfolio analytics:

1. Time-series: A single variable is captured over a period of time, such variances in planned portfolio spend or benefits over a 3-year period. A line chart may be used to demonstrate the trend.
2. Ranking: Categorical subdivisions are ranked in ascending or descending order, such as a ranking of cost/benefit or contribution to strategy (the measure) by projects in the portfolio (the category, with each sales person a categorical subdivision) during a single period. A bar chart may be used to show the comparison across the projects.
3. Part-to-whole: Categorical subdivisions are measured as a ratio to the whole (i.e., a percentage out of 100%). A pie chart or bar chart can show the comparison of ratios, such as the incidence of risk types across the portfolio, or the total portfolio spend or expected benefits by overall health indicator.
4. Deviation: Categorical subdivisions are compared against a reference, such as a comparison of actual vs. budget expenses for enterprise, divisional or department portfolios for a given time period. A bar chart can show comparison of the actual versus the baseline amount.
5. Frequency distribution: Shows the number of observations of a particular variable for given interval, such as the number of years in which the stock market return is between intervals such as 0-10%, 11-20%, etc. A histogram, a type of bar chart, may be used for this analysis. A boxplot helps visualize key statistics about the distribution, such as median, quartiles, outliers, etc.
6. Correlation: Comparison between observations represented by two variables (X,Y) to determine if they tend to move in the same or opposite directions. For example, plotting project size (X) and the incidence of cost and schedule overruns (Y) for a sample of months. A scatter plot is typically used for this message.
7. Nominal comparison: Comparing categorical subdivisions in no particular order, such as project performance by type, size or business division. A bar chart may be used for this comparison.
8. Geographic or geospatial: Comparison of a variable across a map or layout, such as the portfolio performance by office location. A cartogram is a typical graphic used.

Using portfolio infographics

Portfolio infographics can demonstrate critical insights in an instant that are otherwise buried in the data:

• What types of risk are the most common across the portfolio?
• Is there a particular type of resource bottleneck that presents regularly? 
• Is vendor management a recurring problem?
• Is there evidence that project managers are consistently demonstrating forms of cognitive bias in their estimation, such as being overly optimistic?
• Are there particular “break points” in the portfolio (ie, projects or programs upon which there is a high degree of dependency by other projects and programs)? What is the health of the most significant break points? What is the combined value of 'at risk' investment dollars or benefits dependent on these break points? 
• How does the health of tier 1 projects and programs compare with less critical projects and programs?
• Is there a point of increasing complexity at which portfolio performance and throughput tends to 'fall off a cliff'? Is the organisation allowing sufficient schedule and budget contingency in its tier 1 projects for these realities? 
• What is the combined value of investment dollars or benefits by overall traffic light rating? Is the position improving over time? 

Benefits of portfolio intelligence

This information is far more valuable than a list of projects and programs with health indicators. It facilitates evidence-based decision making for steering the portfolio and resolving problems relating to capability and capacity, and provides the business case for action to address systemic problems. This is a more appropriate focus for portfolio governance boards.


The power of this information in the hands of senior management is significant. It can be applied to everything from which risk categories to address first, to where to focus recruitment efforts, to how to tailor project management training needs to get the most value for the organisation.

Roles

This has implications for PMOs. In particular, the role of the portfolio analyst needs to become more analytical and versed in data visualisation techniques to be able to surface key messages and trends out of the detail of portfolio data. In most cases, PMO personnel come from a delivery background, and adopting portfolio analysis and visualisation techniques and approaches requires a significant degree of upskilling.


This in turn highlights the importance of the role of independent portfolio assurance. Assurance, through health checks and other reviews, of your most important projects and programs remains essential. However, a portfolio intelligence expert can give you immediate access to the relevant analytics and visualisation skills needed to bring your portfolio information to life. Such services are a highly cost effective use of your assurance investment, resulting in improved portfolio performance and knowledge transfer.

You can get the full presentation here.

Portfolio intelligence: analytics and visualisation

Written by Bryan Fenech, Director - PPM Intelligence.

Charles Minard's Visualisation of Napoleon's Retreat from Moscow

More often than not the implementations of portfolio management that I come across do not deliver on their promised benefits. The reason for this is almost always the same: portfolio governance boards and portfolio management offices (PMOs) do not make sufficient use of the plethora of data available to them to support decision making about portfolio selection, monitoring and controlling.

Organisations are particularly weak in the areas of portfolio performance, risk and benefits management. The most common approach followed when portfolio governance boards review their inflight portfolio is to run through a list of projects and programs with health ratings (eg traffic lights – red, amber, green) for a variety of performance indicators (eg overall, budget, schedule, resources, risk, and benefits), and some basic commentary. If you are doing this you are wasting your time.

Firstly, it is ineffective. This approach is incapable of creating learning and facilitating decision making that improves structural and systemic problems and inefficiencies regarding the organisation’s portfolio management capability and capacity. If something is going wrong with one of the projects or programs in the list the best that the portfolio governance board can do is authorise actions to be taken by, or with respect to, the project or program concerned. But, secondly and worse still, this is merely duplicating effort that has already taken place at Steering Committee level; the portfolio governance board is doing little more than delegating back (with interest) what the Sponsor has escalated up to them and is already doing their best to deal with. This is not a good use of anyone’s time and energy.

Portfolio Analysis

An alternative approach is to give your portfolio data to a data analyst with project management knowledge and ask them to turn it into valuable information that tells you something about portfolio performance, risk profile and benefits position as a whole, rather than project by project. Here are a few examples of what you might ask them to provide you:

• What types of risk are the most common across the portfolio? Is there a particular type of resource bottleneck that presents regularly? Is vendor management a recurring problem? Is there evidence that project managers are consistently demonstrating forms of cognitive bias in their estimation, such as being overly optimistic?
• Are there particular “break points” in the portfolio (ie, projects or programs upon which there is a high degree of dependency by other projects and programs)? What is the health of the most significant break points? What is the combined value of 'at risk' investment dollars or benefits dependent on these break points?
• How does the health of tier 1 projects and programs compare with less critical projects and programs? Is there a point of increasing complexity at which portfolio performance and throughput tends to 'fall off a cliff'? Is the organisation allowing sufficient schedule and budget contingency in its tier 1 projects for these realities?
• What is the combined value of investment dollars or benefits by overall traffic light rating? Is the position improving over time?

The information derived can be a snapshot at a point in time or historical.

Portfolio Visualisation

This information is far more valuable than a list of projects and programs with health indicators. It facilitates evidence-based decision making for steering the portfolio and resolving problems relating to capability and capacity, and provides the business case for action to address systemic problems.

This is a more appropriate focus for portfolio governance boards.

However, it is critical that portfolio analytics are presented in a way that decision makers can make immediate sense of them. Data visualisation techniques can help to present an incredibly rich picture of your project portfolio. Bringing together graphical representations of portfolio information in a portfolio info-graphic facilitates immediate practical application by senior decision makers. This approach delivers vastly superior outcomes in terms of the quality of learning and decision making.

The power of this information in the hands of senior management is significant. It can be applied to everything from which risk categories to address first, to where to focus recruitment efforts, to how to tailor project management training needs to get the most value for the organisation.

Roles

This has implications for PMOs. In particular, the role of the portfolio analyst needs to become more analytical and versed in data visualisation techniques to be able to surface key messages and trends out of the detail of portfolio data. In most cases, PMO personnel come from a delivery background, and adopting portfolio analysis and visualisation techniques and approaches requires a significant degree of upskilling.

This in turn highlights the importance of the role of independent portfolio assurance. Assurance, through health checks and other reviews, of your most important projects and programs remains essential. However, a portfolio intelligence expert can give you immediate access to the relevant analytics and visualisation skills needed to bring your portfolio information to life. Such services are a highly cost effective use of your assurance investment, resulting in improved portfolio performance and knowledge transfer.

For the purposes of this article I have created some simple visualisations of risk and benefits data using a basic tool. But there isn’t one size fits all for this and there are very powerful tools available to help you explore your data with greater precision and expressiveness. While there are common techniques, it is worth developing your own rich visual representations of your portfolio data. This information is what is unique to your business and it can be enriched so that it becomes one of your most valuable information assets and a source of competitive advantage.

To kill a project ...

Written by Bryan Fenech, Director - PPM Intelligence.

"Action expresses priorities" - Mahatma Gandhi

When to kill off a project, and the decision criteria for doing so, has been a prominent discussion topic among my colleagues lately. It seems that it doesn't happen nearly as often as one would expect. And, perhaps surprisingly, there is not a lot of science that goes into such decisions.

What the theory says

The theory says that you should kill a project when its business case no longer makes sense. There are 2 reasons why this may be the case. Either:

1. Costs are higher than expected because of delay, poor estimation of effort, error and rework, or resources have become more expensive than planned, or
2. Benefits have been compromised due to delay, overly optimistic estimation of new revenue or cost savings, competitors getting to market first, downturn in market conditions, or disruption of the market due to external conditions.

The availability of other projects that represent a better return on the investment of organisational resources should be a determining factor also. Why persist with a project that is no longer expected to deliver valuable outcomes when there are other more attractive options?

What happens in practice

In practice these principles are a good basis on which to make a case to wind up a failed project. More often than not, however, I find its much simpler than that and if there is a major project failure - usually a significant delay or cost blow out - no one will want to be associated with it.

However, statistical studies such as the CHAOS Report suggest that, while about 25% of projects are failures that are killed off at some point, about 50% continue on through to completion even though they are on average around 200% over budget or late. The latter figure suggests its harder than expected to kill a project, and not always a rational decision.

Human factors

This accords with feedback from some of my colleagues from the MBITM Program at the University of Technology, Sydney who work in the field. Consider the following 2 quotes which I think put the matter succinctly:

"It seems to me that many projects don't refer back to the Business Case often enough. And that human trait of wanting to salvage something (anything!) from the investment in the projects leads to a reluctance to kill it off." 

"I've not seen any project 'killed' in my experience. What I've experienced is that when a project fails to deliver, rather than killing it then and there, more resources (money, labour etc.) are thrown at it to 'make it work' - nobody wants to be seen or associated with a failed project. Or it is re-scoped to such an extent that it really bears no resemblance to what was originally envisaged or sign(ed) off on. Successful projects have many parents, while a failed project is an orphan."

Killing a healthy project

I have heard it sometimes said that the maturity of an organisation's portfolio management capability can be measured by its ability to kill projects that are not troubled.

The reasoning is that an organisation with a high portfolio management maturity level will be so in tune with its resource-supply/project-demand equation that it will be able to respond to new opportunities and make space for them by stopping or deferring less valuable in-flight projects.

But is it really that easy? In my experience, not only do organisations not have the tools to do this with any level of rigor, but there is no agreement on the critical issues involved.

For example, when comparing the value of a new opportunity with an in-flight project should we take into account the sunk cost expended to date in the latter? If we do a straight NPV comparison at the time of decision the in-flight project has already expended some of its costs so its NPV at that point will be hard to beat? Alternatively, should we include this sunk cost in the NPV equation? But if we do that then it is no longer an NPV that we are measuring but a present value of both past and future cashflows.

Another problem is that the estimates for in-flight projects are likely to be more accurate than those for a new idea. Should we discount estimates, or apply a risk factor, to account for the different levels of confidence?

What does best practice say?

There are no hard and fast rules here. Both of the key industry standards - Management of Portfolios (Axelos) and The Standard for Portfolio Management (PMI) - are silent on these questions. This is curious given that killing off projects when appropriate is often cited as one of the key objectives of portfolio management.

This area is ripe for field research to survey and gather data on how organisations are approaching these issues.

Portfolio risk management – do you have the right focus?

Written by Bryan Fenech, Director - PPM Intelligence.

“Without it [risk management], portfolio management is just a way to organise the view of projects that will certainly fail” – Scott Berinato in CIO July, 2003.

Portfolio risk management is important; if we characterize an organization's projects as an interrelated portfolio of investments then we need a corresponding portfolio risk management process. This has been borne out by various studies, such as the Standish Group’s CHAOS Report, which highlight a persistent trend of high project failure rates.

Over the course of my career I have come across many ineffective portfolio risk management approaches. The most common problem is that risk management at the portfolio level simply duplicates what is being done at the project and program level. By this I mean that the Portfolio Board or Governance Committee reviews a consolidated list of risks (and their treatments) which have already been reviewed by Project and Program Steering Committees, and which are being managed at that delivery level. This is generally wasted effort because it rarely adds value. More importantly, it is a missed opportunity for the organisation to leverage the advantages and value that a portfolio perspective can bring.

Here is an example to illustrate the point. Imagine we are reviewing a consolidated list of project and program risks as members of a Portfolio Board. Very sensibly we focus our attention on risks that have the potential to derail projects that either have the highest spend or from which the greatest benefits are expected to be derived. However, it may be that the greatest threat to these projects comes not from these risks but from risks impacting other projects upon which they have a logical dependency. Or it could be that the combined impact of risks impacting a cluster of lower credentialed inter-dependent projects is more significant in terms of value at risk. We are failing to incorporate into our risk management approach the view of inter-project dependencies that a portfolio perspective can provide. We are running blind and taking a sizable gamble.

Applying a threshold – e.g., only “catastrophic” and “very high” risks are reviewed by the Portfolio Board – is worse still as this is likely to further obscure the significance of inter-dependencies.

In my opinion, portfolio risk management primarily needs to focus on 3 areas:

1. Investment at risk
2. Common risks, and
3. Domino risks.

Investment at risk

Investment at risk is a measure of the number of projects or the dollar value of projects by risk level. Figure 1 provides a graphical depiction of this using a Red-Amber-Green scheme for risk level.

Figure 1

While this seems like a very simple thing to do it is powerful. For example, where investment at risk is high it indicates that the Portfolio Board may need to pause the introduction of new projects and/or revise benefits and cashflow projections.

Common risks

Common risks are categories of risk that occur most frequently across the portfolio. Figure 2 provides a graphical depiction highlighting the incidence of red ratings by category.

Figure 2

Risks that are common to (or similar across) more than one project or program should receive priority attention because resolving them will have a greater positive impact on overall levels of risk and because they can be dealt with together.

Domino risks

Domino risks are risks that, due to dependency relationships, may have a flow on impact across multiple projects. The things to look for here are:

1. Measuring aggregate value – i.e., the aggregate value, in terms of costs and benefits, of clusters of interdependent projects could be more significant than even the highest priority projects and attention should be focused accordingly
2. Identifying portfolio breakpoints – i.e., projects with the highest number of dependencies with other projects need the most attention because they may take down a significant number of other projects if they fail.

Figure 3 highlights how our priority focus for risk management might change when we incorporate a view of the aggregate value of clusters of interdependent projects.

Figure 3

Key portfolio risk management themes

The key takeout here is that portfolio risk management is about identifying threats to overall portfolio performance and benefits. It complements and uses as an input the risk management activity that is undertaken at the project level. But it needs to mine that information and look for patterns that have portfolio level significance.

Build your portfolio management capability cost effectively with this exceptional resource.

Register NOW at www.udemy.com/project-portfolio-management-essentials.