Stupid Data Miner Tricks

This collective web thing actually works. I got an email from ace quant investment manager John Bogle who'd seen a post from  Paul Kedrosky.  Both were looking for a copy of Stupid Data Miner tricks, a paper in the current Journal of Investing.

The JoI is not fully onboard the "information wants to free" train, so as a good citizen of the interweb series of tubes, I'm depositing an earlier version right here. Download dataminejune_2000.pdf

Here's the introduction:

Disraeli's warning that "there are three kinds of lies: lies, damn lies and statistics"  is particularly true when too much computation is applied to too little data. This paper presents some egregious yet instructional examples of data mining, and describes ways to avoid similar mishaps.

It started out as a set of joke slides showing silly spurious correlations over ten years ago. These statistically appealing relationships between the stock market and diary products and third world livestock populations have been cited often, in Business Week, the Wall Street Journal, the book “A Mathematician Looks at the Stock Market”, and many others. Students from Bill Sharpe’s classes at Stanford seem to be familiar with them. This was expanded, to have some actual content about data mining, and reissued as an academic working paper in 2001. Occasional requests for this arrive from distant corners of the world. So I’d like to thank the editors of the Journal of Trading for publishing this.

Without taking a hatchet to the original, the advice here is still valuable, perhaps more so, now that there is so much more data to mine. Monthly data arrives as one data point, once a month. It’s hard to avoid data mining sins if you look twice. Ticks, quotes, and executions arrive in millions per minute, and many of the practices which fail the statistical sniff tests for low frequency data can now be used responsibly. New frontiers in data mining have been opened up by the availability of vast amounts of textual information. Whatever raw material you choose, fooling yourself remains an occupational hazard in quantitative trading.

PS - DIY dataminers will want to check this: http://swivel.com/

 

Comments

[Soren]

Fun post. Another great link for the DIY data miner is RadipMiner (formely YALE). A fantastic open source java data mining package. Available here: http://www.rapid-i.com/

[Tom the Burninator]

I use the last version YALE, for all my data mining, neural net analysis. It's a rough learning curve but well worth it!

[buy propecia]

What a silly trick .. but very effective jajaja thank you.

[generic cialis]

I'mm 22, and love movies, but there are a lot of classic movies and other really popular movies that I haven't seen. I just signed up for Netflix and want to expand my movie repitoire. Any suggestions?

[Round and Brown]

there are a lot of classic movies and other really popular movies

[costa rica investments]

Thanks for all the great info! I'm still new to Twitter and this was so helpful! I was wondering how it works when you want to respond to something somebody said.

[Wall Street Tour]

"I use the last version YALE, for all my data mining" - this one here is gold for me. Never thought of that. THanks

[viagra online]

Oh man, how could you not have put down "Amelie"? I watch that over and over again for her hair...her clothes...and just the basic richness of the movie.

[air jordan shoes]

To make an airplane contrail, you need water and some kind of microscopic particle for the water to condense around.

[vibram five fingers]

These statistically appealing relationships between the stock market and diary products and third world livestock populations have been cited often

[dr dre headphones]

Thanks for all the great info! I'm still new to Twitter and this was so helpful! I was wondering how it works when you want to respond to something somebody said.

[Nike Lebron Shoes]

http://www.lebronjameswebsite.com/ They're so charming in the athlete square. People would prefer to choose up a unique color for himself to show himself.

[Dating sites reviews]

"I use the last version YALE, for all my data mining" - this one here is gold for me. Never thought of that. THanks and good luck!

[cialis to buy]

Cialis cheap prices http://www.maxipharmacy.com/ pregnancy cialis.

[fashion jewelry online]

It may be accept by citizens so effortlessly. I assume some people will agree with you. That you're perfect that we are able to to not rely on other people. Who we are able to to rely on is ourself.

[vibram five fingers]

These statistically appealing relationships between the stock market and diary products and third world livestock populations have been cited often

[Kamagra]

I wonder how you got so good. This is really a fascinating blog, lots of stuff that I can get into. One thing I just want to say is that your Blog is so perfect!..

[Kamagra]

Skönt att vara besöka din blogg igen, har det varit månader för mig. Väl här artikeln som jag har väntat så länge. Jag behöver den här artikeln för att slutföra mitt uppdrag i kollegiet, och den har samma tema med din artikel. Tack, bra aktie

[office 2007 key]

I just signed up for Netflix and want to expand my movie repitoire. third world livestock populations have been cited often.

[cialis to buy]

drug cialis tadalafil http://www.maxipharmacy.com/ cialis 5mg cialis levitra pharmacycom cialis vs fluorouracil.

[Phentermine]

why???

[Mobile Computing]

Wow, Great post,Nice work, I would like to read your blog every day Thanks

[ザイロリック]

Great tips, I would like to join your blog anyway,

[write my paper]

"I use the last version YALE, for all my data mining" - this one here is gold for me. Never thought of that. THanks and good luck!

[SEO Services India]

There are may person searching about that now they will find enough resources by your post..