METADATA AND DATA MINING: TWO UNLIKELY PEAS IN A POD, PART I
by W H Inmon

The relationship between data and data mining is clear. Data mining cannot be done unless there is data. But the relationship between metadata and data mining is not nearly so clear. Although it is hardly obvious, there is a strong and important relationship between the activity of data mining and metadata.

WHAT IS METADATA?

What is metadata? Metadata is data about data. Metadata describes various aspects of data. Some simple examples of metadata are

• structural information. How is data stored and organized,
• metric information. Information about how much data there is and how it is distributed,
• business information. Information about how data is used in business and what its non technical definition is, and so forth.

Metadata then is incidental or indirect data that describes one or more aspects of a data base.

The relationship between data mining and metadata at first glance seems to be far fetched. But in actuality, the relationship is very strong, although the relationship is not immediately obvious. In order to explain why there is indeed a close relationship between metadata and data mining, consider historical data.

HISTORICAL DATA

Historical data makes up the backbone of the data warehouse environment. Depending upon the definition of what is historical and what is current, historical data may constitute up to 99% of a data warehouse.

But historical data is also intensely interesting to the data miner. It is historical data that provides the basis for finding

• patterns,
• trends,
• relationships,
• associations, and so forth.

Not only does historical data provide the foundation for discoveries, it is historical data that allows the changes in time that measure those discoveries to be quantified. In short, the data miner cannot live without historical data.

EVER CHANGING HISTORICAL DATA

But historical data is a peculiar beast in that it gives the appearance of being static. Instead, once examined closely, it is seen that historical data is constantly changing.

If an analyst looks at data at any one point in time, the form and structure of the data appears to be static. At any one moment in time there is

• a single structure of data,
• a single definition of data,
• a single value of data, and so forth.

But viewed over time the data is seen to be anything but static.

Over time many aspects of data change. Some of the aspects of data that change over time include

• structure. One day a record of data has one structure; another day the structure of data has changed,
• meaning. One day there is one definition for what a product is; the next day the definition of what a product line is changed,
• classifications. One day a Canadian sale is classified as a North American sale; the next day a Canadian sale is no longer a North American sale,
• calculation. One day the tax rate is 32% for a given bracket; the next day the tax rate is 36%,
• economic conditions. One day the employment rate is 4.1%; the next day the rate is 5.6%.

For more information, see http://www.pine-cone.com