• :

    Machine learning (ML), calibration of mathematical models using experimental data, has enabled tremendous advances in computational science. Some such advances have included digital character recognition, diagnoses from medical imaging, and forensics. However, models developed from ML are often treated as a black box, where data goes in and predictions come out. Very little concern is given to the uncertainty in the predictions of these models, so consequently not very much can be said about their reliability. This becomes of critical concern when dealing with questions that may have life-altering consequences. The central issue is whether life-and-death decisions can turn on the result of a model that no one understands. Knowing the uncertainty in a ML model’s predictions allows for more robust assessments of the model’s performance and can even provide some insight into its underlying behavior. Understanding and appreciation of uncertainty in models would be improved if modeling packages had a more transparent means of estimating uncertainty. Several computational methods have been developed to estimate uncertainty in ML models, including Bayesian uncertainty analysis and bootstrapping. Likewise, these algorithms have been implemented in various computer packages. In this talk, I will discuss existing uncertainty estimation codes, with focus on the strengths of their implementation and potential areas for improvement. I will focus on NIST’s MUM-PCE as well as a machine-learning uncertainty package under development at NIST, although the conclusions that I draw will be broadly applicable to most uncertainty analysis packages. In general, although uncertainty analysis codes serve well at implementing the uncertainty analysis algorithms, they are still mostly written for uncertainty experts rather than users of common simulation packages.

  • : David A. Sheen
  • : National Institute of Standards and Technology
  • : David A. Sheen
  • : big_data
  • : intermediate
  • : david.sheen@nist.gov
  • : 301 975 2603
Automated Uncertainty Analysis for Model Calibration and Machine Learning