SENSITIVITY ANALYSIS: INSIDE THE BLACK BOX
by Mike Niemann

As analysts have become more acquainted with neural network based prediction, the method's critics have become less vocal. However, the "black box" image of neural networks regarding the results they produce still persists. This perception remains primarily because the one-to-one relationships between independent variables and regression coefficients of traditional statistical methods have been replaced by a complex myriad of weights in neural network models.

Critics argue that variable relationships are extremely difficult to deduce from such networks of model weights and, hence, call the inner working of the network a "black box". Fortunately, alternative statistical measures exist that can help analysts evaluate the predictiveness of independent variables in neural network models. This article analyzes the insight that sensitivity analysis can provide.

Since regression analyses are the entrenched standard, beta (?) coefficients and their corresponding test statistics are the typical metrics reported for independent variable evaluation. For example, a simple linear regression equation might appear as follows: Y = ? + ?X, where Y is the dependent variable, X is the independent variable, and ? and ? are parameters used to help estimate levels of Y from levels of X. The value and sign of ? tells the analyst the nature of X's relationship to Y. Furthermore, a significance test can be used to determine if the effect of X on Y is statistically significant. This involves calculating a statistic to test the hypothesis that ?'s value is actually equal or extremely close to zero (?=0 demonstrates the lack of a relationship between the two variables). Beta weights and test statistics lend themselves to easy interpretation and provide meaningful information about variable relationships. Since neural networks lack easily defined coefficients, different metrics relating to variable strength and validity must be applied.

Variable sensitivity analysis utilizes partial derivative information to determine which independent variables are most "sensitive" with respect to affecting a change on a dependent variable. A derivative essentially calculates one variable's rate of change with respect to another. When trying to understand variable predictiveness, there is a simple question to be addressed: when manipulated, which independent variables produce the greatest change in the dependent variable. By calculating the derivatives for all patterns of input-output data pairs in a dataset, the impact of the independent variables with respect to predicting the dependent variable can be evaluated.

Sensitivities not only identify the magnitude of a relationship between variables but also whether that relationship is positive or negative. Sensitivity analysis does lack the statistical "rule of thumb" for variable selection which test statistic scores provide in regression analysis. However, sensitivity analysis conveys a robust means of comparison for the evaluation of one independent variable's predictive strength relative to another. Sensitivity analysis is only one of many analytic tools that are helping to remove the "black box" stigma from neural network based predictive models.

---

For more information, see http://www.trajecta.com