Credibly reaching a reliability target using a model initially constructed by expert elicitation
- 1k Downloads
The Defense Advanced Research Projects Agency Defense Science Office (DARPA/DSO) is sponsoring Open Manufacturing (OM), an initiative to develop new technologies, new computational tools, and rapid qualification to accelerate the manufacturing innovation timeline. Certification Methodology to Transition Innovation (CMTI), an OM program, has developed a methodology to quantify the effect of manufacturing variability on product performance to address the risk to cost and performance associated with failure to take manufacturing capability and material and fabrication/assembly variation into account early in the design process. An important aspect of this program is the use of Bayesian networks (BN) to evaluate risk. The BN is used as a graphical representation of the contributing factors that lead to manufacturing defects. The reliability of the final product is then analyzed using the contributing factors. There are many types of programs where there is little relevant data to support the probabilities needed to populate the BN model. This is very likely the case for new programs or at the end of long programs when obsolescence challenges servicing a product when original vendors are no longer in business. In these cases, probabilities must be obtained from expert opinion using a technique called expert elicitation. Even under objective ‘Good Faith’ opinions, the expert himself has a lot of uncertainty in that opinion. This paper details an approach to obtaining credible model output based on the idea of having a hypothetical expert whose unconscious bias influences the model output and discovering and using countermeasures to find and prevent these biases. Countermeasures include replacing point probabilities with beta distributions to incorporate uncertainty, 95% confidence levels, and using a multitude of different types of sensitivity analyses to draw attention to potential trouble spots. Finally, this paper uses a new technique named ‘confidence level shifting’ to optimally reduce epistemic uncertainty in the model. Taken together, the set of tools described in this paper will allow an engineer to cost effectively determine which areas of the manufacturing process are most responsible for performance variance and to determine the most effective approach to reducing that variance in order to reach a target reliability.
KeywordsCredibility Expert elicitation Confidence level shifting Monte Carlo Uncertainty quantification Targeted testing Unitized testing Uncertainty reduction Epistemic uncertainty Reliability targets
delta, change in value
beta distribution parameter expressing the number of flawed examples
beta distribution parameter expressing the number of flawless examples
confidence level shifting
Certification Methodology to Transition Innovation
Defense Advanced Research Projects Agency Defense Science Office
global sensitivity analysis
expert confidence in estimate in terms of equivalent prior sample size
the most likely probability of a flaw
negative result test
proportion of flaws in the beta distribution
probability of node i inducing or failing to detect a defect
probability of failure
sensitivity of the 95% CL of model output due to change in mode of node i
derivative-based sensitivity measure
effect due to variable i
total effect due to variable i
This paper is sponsored by Defense Advanced Research Projects Agency, Defense Sciences Office under the Open Manufacturing Program, ARPA Order No. S587/00, Program Code 2D10, issued by DARPA/CMO under contract no. HR 0011-12-C-0034. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressly or implied, of the Defense Advanced Research Projects Agency of the U.S. Government. This paper was approved for public release, distribution unlimited as 14-00070-EOT.
- 1.Renieri G: High performance stiffened panel design concept. Annual Report. Office of Naval Research (ONR), Arlington, VA; 2013.Google Scholar
- 2.Koller D, Friedman N: Probabilistic graphical models: principles and techniques. Press, MIT, Cambridge; 2009.Google Scholar
- 3.Hahn GL, Pado LE, Thomas MJ, Liguore SL: Application of risk quantification approach to aerospace manufacturing using Bayesian networks. Paper presented at AIAA SciTech 2014. National Harbor, Maryland; 2014.Google Scholar
- 4.Saltelli A, Ratto M, Andres T, Campolongo F, Cariboni J, Gatelli D, Saisana M, Tarantola S: Global sensitivity analysis. The Primer, Wiley, John & Sons, , Incorporated; 2008.Google Scholar
- 5.Kruschke J: Doing Bayesian data analysis. Press, Academic, New York; 2011.Google Scholar
- 6.Kruschke J: Beta distribution parameterized by mode instead of mean. 2012.Google Scholar
- 7.Haldar A, Mahadevan S: Probability, reliability, and statistical methods in engineering design. Wiley, John & Sons, Incorporated, New York; 2000.Google Scholar
- 8.Milton JS, Arnold JC: Probability and statistics in the engineering and computing sciences. Elsevier Academic Press, Burlington, MA; 1986.Google Scholar
- 9.Abrahamson N: Seismic hazard assessment: problems with current practice and future developments Proceedings. First European Conference on Earthquake Engineering and Seismology, Geneva, Switzerland; 2006.Google Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (https://creativecommons.org/licenses/by/4.0), which permits use, duplication, adaptation, distribution, and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.