HISTORICAL DATA: THE FOUNDATION OF DATA MINING, PART II
by W H Inmon

The changing nature of data, especially at the detailed level, presents an intriguing story when viewed over time.

RESTATING DATA OVER TIME

The classical approach to the amendment of data over time (or the restatement of data over time) is to go back and alter the detailed data, and do a recalculation based on the amended data.

The process to do restatement at the detailed level is tedious and extremely resource intensive. In every case. But when old detailed data is examined and restated it can be relied upon to create a basis for current and crisp analysis and calculation. The data miner feels very satisfied once the old detailed data has been restated properly.

SOME DRAWBACKS TO RESTATEMENT

But going backward in time and reshaping the detailed data has a surprisingly large number of drawbacks. Some of the drawbacks with retro fitting data are -

• the resources required. The data needs to be examined at the most detailed level. To do so requires many machine resources for reading and writing large amounts of data,
• once restatement of detailed data is done, reconciliation of any processing done against the data prior to restatement is impossible,
• restating data can be a very complex process,
• in some cases restatement is logically inconsistent. For example, if the Post Office creates zip code 80111 in 1997, does it make sense to create sales for that zip code in 1996? Representing a sale made for zip code 80111 in 1996 is an artificial thing to do since zip code 80111 did not exist in 1996,
• some restatements are very temporary affairs. Once the data has been restated, it is predictable that the analyst will want to restate the data again, but in a different manner.

There are then some powerful reasons not to restate historical data at all. Indeed, at some point in time the volume of data that must be passed during the restatement process precludes historical restatement.

NOT RESTATING DATA

What does an organization do when the point is reached where it is impractical to restate data? There are several possibilities.

The first possibility to make best of the circumstances of not restating data is to make extensive use of metadata. By using metadata to carefully track and define data over time, and to use the metadata to track the changes to data over time, the analyst is able to interpret the changes in data correctly.

A second approach is to operate at a summary level. When examined closely, the differences in data over time are most relevant to detailed data. At the summary level the historical discrepancies that occur are "ironed out". In other words, the higher the level of summarization, the less problem there is with historical restatement.

So simply ignoring historical differences in data at the detailed level and not restating the data is an acceptable approach in many cases.

PRECEDENTS

Is there a precedent for allowing historical discrepancies to occur and stand with no correction or restatement? Interestingly, there are many valid and normal cases. Consider the following -

• the measurement of GNP from 1900 to 1999. The definition and calculation of GNP or GDP has changed many times over the years. But the general demeanor of the calculation remains constant,
• the statement of corporate revenue from 1920 to 1999. The tax rules and classifications have changed many times over the years. But the amounts made by a corporation remain a single important measurement. The fact is that the measurement of revenue of 1942 is not at all the same measurement of revenue of 1978. But at a summary level the differences in the basis of calculations is of little concern,

There are then many measurements over time that exist where the basis for the measurement has changed. The notion that data MUST be classified on exactly the same foundation over all units of time is an idealistic one, but is impractical for all but the smallest collections of historical data.
---
For more information, see http://www.pine-cone.com