Through the State Highway Operation and Protection Program (SHOPP), the California Department of Transportation (Caltrans) was investing around USD 2.3 billion a year at the time in maintaining, rehabilitating and improving the safety of state highways and bridges. In September 2014, Ralph Keeney and I advised Caltrans on how to prioritise these projects more transparently.
Client: California Department of Transportation (Caltrans), SHOPP Pilot Project
Period: September 2014
Role: Advisory work in the framing phase, together with Ralph Keeney
Output: Recommendations report “Recommendations about Decision-Making for Caltrans SHOPP Project Prioritization” (Keeney & Siebert)
Method: Value-Focused Thinking, Multi-Objective Decision Analysis
The starting point
Until then, SHOPP projects were prioritised through 31 programmes. Whether a project received funding depended mainly on which programme it was assigned to and how highly that programme ranked. What a project actually achieves – across several assets and relative to its cost – could hardly be captured this way. Caltrans had just adopted a new vision, mission and set of goals and had compiled an extensive list of possible evaluation criteria. The question was: how do you turn this into a prioritisation that is logically sound and can be justified to policymakers and the public?
What we did
On 22 September 2014, Ralph Keeney and I held a series of meetings in Sacramento over the course of one day: with the core project team, with the heads of finance, project delivery, maintenance and planning, with two district directors and with a representative of the California Transportation Commission. We wanted to understand how SHOPP decisions were being made and what leadership hoped for from a better process. We then summarised our recommendations in a report. Three of them I still consider central.
Fundamental objectives instead of criteria lists
Most of the criteria collected were means to an end rather than actual objectives. An example:
Improve pavement → avoid accidents → save lives
Only the last link is a fundamental objective – something we care about for its own sake. Evaluating means objectives and fundamental objectives together counts the same benefit twice. We did not have to invent the fundamental objectives from scratch: they could be derived directly from Caltrans’ new goals – maximise safety, minimise costs, minimise disruption of the economy and minimise inconvenience. Two further Caltrans goals, system performance and organisational excellence, depend more on the agency’s actions as a whole than on the selection of individual projects and therefore do not belong in the project evaluation.
Making objectives measurable
Each fundamental objective needs an attribute that measures how well an alternative achieves it. Sometimes there is a natural measure, such as cost in dollars. Sometimes a proxy measure has to do, such as accidents avoided instead of fatalities avoided. And sometimes it is worth constructing a dedicated measure that, for example, weights head-on collisions more heavily than minor accidents.
Weights from trade-offs, not from ‘importance’
The question ‘Which is more important, accidents or costs?’ has no meaningful answer. Is one avoided accident more important than a billion dollars? Hardly. More important than a thousand dollars? Probably. The only meaningful question is how much a specific change in one objective is worth compared with a specific change in another. If, for instance, one avoided accident is valued at two million dollars, a project that costs 120 million dollars and avoids 100 accidents creates a net value equivalent to 80 million dollars. Value judgements of this kind can be made explicit, discussed and checked against the organisation’s mission.
What I took away
The greatest leverage lay not in the calculation but in the question that comes before it: what does the organisation actually want to achieve with its projects? Once the fundamental objectives are clear, data requirements, weighting and communication become much easier. Good prioritisation starts with objectives, not with data.
Postscript (September 2026): Building on our recommendations and further advice from Lee Merkhofer, the Caltrans team developed a prioritisation model based on Multi-Objective Decision Analysis, including a prototype, by June 2015. The Caltrans goals from which we derived the fundamental objectives in 2014 remained the agency’s official goals until 2020. Since then, Caltrans has reformulated them in two strategic plans; safety remains at the top, and economic prosperity is now a goal in its own right. Caltrans has published the final report of the pilot project, with our recommendations report in the appendix: SHOPP Pilot Project – Phase 1: A Framework for Project Prioritization (PDF, appendix).