Validation of Systems Engineering Leading Indicators: A Bayesian Belief Network Approach
Abstract
Systems engineering continues to be a critical area in defense acquisition; multiple reports cite its importance in the success of programs (GAO 2011, GAO 2015). One of the efforts in response to this need was the collaboration between the Army’s Practical Software Measurement group and the Air Force-funded Lean Aerospace Initiative at MIT on the development of leading indicators for systems engineering (2004-2007). Leading indicators are predictive measures which, if used correctly, can provide insight into the health of systems engineering functions in large projects. When applied to systems engineering, leading indicators are mathematical composites of relevant and quantifiable products, project progress, or process attributes (measures) that are taken over time that communicate important information about quality, processes, technology, products, projects, and/or resources (Roedler & Rhodes 2005). The motivation for the development of leading indicators was the release of several policies calling for the improvement of systems engineering processes on programs by the US Department of Defense and the US Air Force in 2003–2004 (Sambur 2003). These policies called for the development of measures for evaluating the goodness of systems engineering processes on a program. As a result, a working group of experts from academia, government, and industry was formed to define measurable systems engineering constructs and interpretation guidance. The result was a list of 13 leading indicators (Rhodes, Valerdi & Roedler, 2009): 1.Requirements Trends 2.System Definition Change Backlog Trend 3.Interface Trends 4.Requirements Validation Trends 5.Requirements Verification Trends 6.Work Product Approval Trends 7.Review Action Closure Trends 8.Risk Exposure Trends 9.Risk Handling Trends 10.Technology Maturity Trends 11.Technical Measurement Trends 12.Systems Engineering Staffing & SkillsTrends 13.Process Compliance Trends While these indicators provide a useful starting point for measuring the effectiveness of systems engineering processes and the likelihood of program success (Figure 1), their validation was limited to a survey at a single defense contractor inquiring about the usefulness of each indicator. Furthermore, the indicators were developed through expert opinion which made it impossible to falsify them. This highlights two limitations of the systems engineering leading indicators: (1) cause and effect relationships between the leading indicators and program success were not validated as a predictive indicator, and (2) the interactions between leading indicators were not explored. For purposes of this project, program success will be defined as an index comprised of compliance to cost, schedule, and performance objectives.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Jan 05, 2017
- Source ID
- N002441710007
Entities
People
- Ricado Valerdi
Organizations
- Office of the Secretary of Defense
- University of Arizona