MODELING FOR MAXIMUM PROFIT
James Lemieux, Senior Analyst, Trajecta, Inc.

One goal in building statistical models for business applications is to give decision makers useable information. However, a statistical modeling method which provides a clearer understanding of these business applications would be helpful. Only in this way will the decision makers receive the information they can truly use to enhance the bottom lines of their companies. This article describes the threshold function, which considers the costs associated with business decisions when determining how to apply a statistical model to a dataset. (Although the threshold function was developed specifically for use with Trajecta's neural network modeling tool dbProphet, the concepts inherent in the function could be applied to any modeling tool.)

Decision makers cannot directly use such traditional measures of model strength as the R-squared measure or the maximum Kolmorgorov-Smirnov value. Although, these are good mathematical measures of model accuracy, they do not determine the best set of customers to target for a marketing campaign. Instead, the decision maker needs a statistical measure incorporating profit and cost information. It is only when the measure takes into consideration the bottom line figures that the models efficacy will be uncovered.

The threshold function created in Trajectas dbProphet is used when scoring a response model to accurately identify the most profitable group of participants to a given marketing effort. This effort could be via direct mail, telemarketing, or any other sales medium which has readily identifiable costs and associated revenues for each offer made. The parameters included in this function are the costs associated with the two types of errors in response models.

A Type 1 error occurs when a model predicts a nonpurchaser as a purchaser. Conversely, a Type 2 error occurs when a purchaser is predicted to be a nonpurchaser. Although, statistically both errors ought to be avoided, they do not impact profitability equally. For example, the cost of a Type 1 error is the price of mailing to or telephoning a nonpurchaser. However, the cost of a Type 2 error is much higher since it represents lost revenue. By not mailing an offer to or calling a person who would have made a purchase, a company loses the opportunity to sell to that customer. The fact that a Type 2 error is actually more costly than a Type 1 error tends to skew traditional statistical measures, such as the lift chart. The example above demonstrates the use of the threshold function and how it compares to a traditional lift chart.

A useful measure of a models performance is often created by first sorting the column containing the actual purchasers/responders in the ascending order of the column containing the predicted purchasers/responders. Good models tend to sort most of the actual responders towards the top of the list. Traditionally, information is gleaned from this sorted list by finding the cumulative percentage of responders above a threshold as it moves downward through the sorted list.

For example, the so-called top decile score of a model is found by calculating the percentage of responders above the threshold marking the top 10 percent of the sorted list. A lift chart is simply a graphical representation of the top decile scores for every 10 percent increment in the sorted list. (see footnote 1)

A common use of the lift charts in a direct mail context is to determine which group (i.e. decile) in the sorted list should be sent mail offers. However, as stated before, the interpretation has a key flaw. That is, no cost information is included in the calculation of the lift chart. The threshold function used in dbProphet takes the interpretation of the sorted list of actual responders one step further by assigning a cost to each row.

The costs are determined as follows. Suppose one wished to send mailing offers to the top ten percent of the sorted list. Although good models will tend to place many actual responders in the top ten percent of the sorted list, not every row will contain an actual responder. One could assign a mailing cost for every row above the threshold line (i.e. in the top ten percent of the sorted list), and assign an expected revenue for every actual responder above the threshold line. (see footnote 2)

In the rows below the threshold line, costs would be assigned only to the actual responders. This cost represents the lost opportunity to attain revenue from potential customers falling below the threshold line. The threshold function is found by simply adding up the revenue terms and subtracting off the cost terms in the above assignment. One can interpret the threshold function in a straightforward manner by simply finding the threshold point yielding the maximum value of the threshold function. Since the function represents profit, it has an immediate use to a decision maker.

There are numerous ways to interpret statistical models. Unfortunately, not all statistical measures were created equal. Some of the most commonly used measures of model accuracy are nearly meaningless when directly applied to business problems. The threshold function was created to remedy this situation, by introducing a statistical measure which reflects the strength of a model and has a useful business interpretation. The modeling community would do well to employ such measures when reporting its findings to management.

Footnotes --
(1) Actually, any incremental percentage of the sorted list can be used. For example, a lift chart can be formed by looking at the top one percent, then looking at the top 2 percent, and so on; recording the corresponding percentage of total responders in each group.

(2) The easiest expected revenue to use is the average revenue for each responder from a previous mailing effort. Alternatively, one could use dbProphet to build a model predicting the expected revenue for each person mailed. This predicted revenue can be used in place of the average revenue and may tend to yield a more accurate representation of the expected profit of a mailing campaign.