Content Matters

Much of the debate the past week or two over the NSA data mining activities seems to be misdirected.

Critics of the program talk about how the Government should not be allowed to use data mining tools to monitor activities of the public.  Meanwhile, defenders of the program fall back on the old “if you don’t allow us to do this, you’re just helping the terrorists”.  Like many debates in Washington these days, this one is unlikely to result in meaningful results unless the focus is shifted.

Data mining is a technology that’s been around for more than two decades.  Ever since the early business intelligence software providers made the “beer and diapers” argument, companies have used data mining to understand customer behavior.  In fact, some of my favorite companies use data mining to improve my customer experience.  For example, Amazon uses data mining to recommend products to me, based upon past purchases. 

The law enforcement community has only recently begun to use data mining.  Sophisticated investigators use link analysis tools such as I2’s Analyst Notebook to help identify connections between people, places and activities.  During the past five years, the intelligence community has begun to explore use of data and text mining applications to improve their results.

In my view, use of data mining tools is a good thing.  Whether you’re a company trying to understand your customers, or a law enforcement agency trying to identify potential terrorists, leveraging technology to automate manual processes is a natural.  But, with these expanded capabilities come new responsibilities.  For corporate citizens, that means transparency.  I allow Amazon to use my purchase and browsing history to make suggestions because they publish their privacy policies and seem to live up to them.  If it were to come out that Amazon were not abiding by their privacy policies (perhaps selling my purchase history to other companies), they would lose the trust of customers and I, for one, would stop shopping there.

In the public sector, transparency is more difficult.  It’s hard to come out publicly and state that we intend to monitor phone traffic to see who’s calling whom.  That could certainly tip off the bad guys (though from everything I have read, the bad guys have long ago begun using internet calling and web tools like the anonymizer to mask their trail).  Rather than absolute transparency, I think that the government needs to demonstrate that it can be trusted.  And, that’s where the problems lie for this administration.  It has been the most secretive of any administration in my lifetime (more Nixonian than Nixon), reclassifying 50-year old documents that were in the public domain, using the courts to keep from disclosing the members of the Vice President’s energy task force and threatening criminal charges against the media.  It also has shown a willingness to violate privacy and national security for political purposes, most notably in the Valerie Plame affair.

I support use of data mining technologies to improve our intelligence.  But, the process requires oversight and, where feasible, transparency.

There’s a similar lesson in here for business.  Your customers will often be willing to share significant information with your company, provided that there are benefits to doing so (either to themselves or to the community as a whole) and that they trust you not to use that information for any other purpose.  Transparency and honesty remain the building blocks of successful information-based businesses.

Footnote: Beer and Diapers
For those unfamiliar with the beer and diapers story, it was the primary example used by sales reps from BI companies for many years.  The premise is that, by mining transaction logs, a convenience store was able to see that on Saturday nights after 10pm, there was a strong correlation between sales of diapers and beer.  In essence, families running out of diapers late on a Saturday night inevitably sent dad out to buy more, and he grabbed a six-pack to go.  In leveraging this information, the convenience store could decide whether to place beer next to diapers (to sell more beer to the diaper dads) or to place them further apart, requiring the dads to go through other aisles, perhaps adding a bag of chips to their purchase.  While the “beer and diapers” story is so oft-told (with many variations) that it has taken on the appearance of an urban legend, it’s origin seems to be based in a true study conducted by Teradata on behalf of Osco Drugs in 1992.