Nerds on Wall Street

Unencumbered by the Thought Process

This is the web edition of Nerd on the Street,
Institutional Investor Magazine, November 2009
(Download the slick print version here)

Trying to understand what happened to capitalism in 2008 without serious data collection and analysis is a poster child example of reasoning “unencumbered by the thought process”,  the theme of a brilliant commencement address given at the Massachusetts Institute of Technology by two smart graduates of that institution, National Public Radio’s Click and Clack, the Car Talk Guys, a.k.a. Tom and Ray Magliozzi, who only pretend to be morons on their show.

Click & Clack (not actual morons), and the UBTTP Flag, in Latin, with tailfins.

Behind their humorous message was a painful truth. We make all too many important decisions without firing up a full set of cerebral pistons. The U.S. Congress seems particularly prone to action without sufficient thought. Consider the House Financial Services Committee. It has — count them — 72 members, enough to field a softball league that could be coached by committee chairman, Congressman Barney Frank. Many are in dicey seats, having been placed on the committee to draw cash from this campaign contribution honeypot.

 

Coach Barney (in a much earlier season).

I'd like to believe that this crowd will use a careful, scientific approach, despite the distractions of health care, two wars and trying to get reelected. I also like unicorns.

The truth is, these public servants, seeking to interpret the greatest crisis since the Great Depression, are doing no more than rounding up the usual suspects — the fat-cat bankers, the slick fund managers and the like — and saying “You’ve been bad!” in front of a battery of cameras. Such theater may be great for scoring quick political points, but it isn’t particularly enlightening.

While Congress dithers, others have put forth some excellent proposals to help now. One comes from Andrew Lo, director of MIT’s Laboratory for Financial Engineering. He suggests a Capital Markets Safety Board modeled after the National Transportation Safety Board. After the next financial crash, “an experienced team of forensic accountants, lawyers and financial engineers [will sift] through the wreckage” of financial institutions and prepare a public report explaining how it all happened. Not only will the study decipher the immediate causes of the crash, it will generate recommendations on how to avoid crashes in the future.

 

If financial systems had flight data recorders, what would they record?
And who would read them when they came back looking like this?

Another fine suggestion comes from the Committee to Establish the National Institute of Finance, a mix of academics and Wall Street sorts who have the wacky notion that this is a complicated business, and that perhaps keeping an eye on things before they crater the economy would be a good idea (sign up at
ce-nif.org).

Either approach — a Capital Markets Safety Board or a National Institute of Finance — would offer a much-needed dose of the scientific method, fully encumbered by the thought process. Let’s not waste this crisis. And don’t drive like my brother.

_____________

Extra Web Goodies & News: 

The SSRN site with the link to Andy Lo's paper is an amazing place. A true testimonial to academic freedom (after tenure anyway) is found in the remarkable top-downloaded paper for much of last year, which has a four letter title that is sure to catch any scholar's attention, particularly thirteen year old boys. 

Harry Markowitz, Nobel Laureate in Economics (1990), is the latest member of the Committee to Establish the National Institute of Finance.

"Ready, Fire, Aim" Market Reform

This is an annotated, linked & expanded version of  Nerd on the Street ,
Institutional Investor Magazine, September 2009

Previous disasters, all notably less damaging than the Great Mess of '08, have led to deep serious analysis of their causes. The general notion being that understanding the causes might be useful in avoiding nasty repeat performances. This has been Standard Operating Procedure for a long time for good reason. The alternative Ready Fire Aim approach can be hazardous to your health, yet that seems to be the plan for the newly announced Financial Crisis Inquiry Commission, tasked with reporting back on the crisis that nearly broke capitalism by the end of 2010, while the legislation to avoid a financial crisis is being crafted now.

There have been more than a few comparisons with the Pecora Hearings in 1933. Pecora was a former New York DA, who as counsel for the Senate Banking Committee, had enough clout, backing and sense of mission that in six intense months, his investigation led to the legislation that established the SEC and the basis for modern market regulation. He also, famously, got J.P. Morgan to admit to not paying taxes for years, and to let a lady circus midget sit in J.P.’s illustrious lap.

Could Angelides get a midget
into Larry Summers’ lap?

A more recent comparison is the Brady Commission, appointed by Ronald Reagan less than three weeks after the crash of October 1987.  Let’s take a Nerd approach and compare a few salient features of the two investigations. The details are found in the "More than you want to know" section at the end of this column. 

(Research assistance from Nina Ji, Institutional Investor. See supplements for details.)

The absence of any sense of urgency in this is odd.  Back in October of 2008, the Heritage Foundation called for a Brady style commission immediately: “ Appointing a commission now would avoid losing the three months until the next President takes office.”

Now, a mere nine or ten months later, the commission has been named, but they appear to be in no great hurry to do anything. “Angelides, a former California state treasurer, said,  .. he hoped to name an executive director "sooner rather than later." But he was unsure this could be done by [the first meeting on] September 17. “   Contrast this with Brady, who named his executive director, Harvard Business School professor Robert Glauber, immediately. Together they assembled an 50 person team, and delivered a report that “surprised even Wall Street experts with its wealth of detail and cogent explanations of what went wrong” . They did this on schedule, in 60 days, less time the current commission is taking to pick a director, let alone a staff.

Ten members on the commission is hardly lean and mean. Brady had four. By all accounts, Phil Angelides is an able public servant, best known as the sacrificial democrat offered up to be whacked by the Gubernator in 2006 (a choice many voters may regret, as the state sinks into the fiscal sea).  But he is not exactly a heavy hitter in the arcane world of modern Wall Street. Brady’s father was a partner of John D. Rockefeller. He knew the markets, and all the players knew that Brady and his team of financial and computer trading experts could not be fooled.

Others on the current commission, while accomplished citizens all, seem distant from the core issues that came close to burying us in toxic assets. A Las Vegas investor, Chairman of an anti-virus firm, a former McCain campaign operative. One member, Brooksley Born, clearly understands what’s going on. As chair of the CFTC in 2000, she was on the losing end of a battle with Rubin, Summers and others about regulating the very activities that caused the crisis. She has “I told you so” bragging rights. We can hope that she and her commissioners have the horsepower to expose what many would still like to hide.

In many investigations, choosing an irrefutable and universally respected leader provides that horsepower. Think of Nobel prize-winning physicist Richard Feynman in the investigation of the Space Shuttle Challenger accident. There are close to a dozen living American Nobel laureate economists, several in finance.  Were any offered a job?

In a parallel bizarro world, maybe we’d have a Volcker-Buffet Commission to figure out what hit us and what to do about it? It was nice to see the wise old guys during the Obama transition. Buffet was mentioned for Sec Treasury, but they've been scarce lately.   Is it too late to actually give them something to do, and hold off on changing the game until they do it?

 

___________________________________________________________________________________

Supplements, bonus stuff and web exclusives!

More than you want to know about the Brady Commission (pdf)

Angelides Commission members & notes (pdf)

Back into the data mine.

Jason Zweig of the Wall Street Journal is the co-author of The Intelligent Investor: The Definitive Book on Value Investing. A Book of Practical Counsel, the one investment book everyone should have. Ben Graham's original version was published in 1949, the new one is co-authored with Warren Buffet, so you know this is quality merchandise. It is straightforward, gimmick-free advice, with no scent of dubious "get rich quick" formulas.

In contrast, a strong contender for the mother of all gimmicky, dubious, get-rich formulas is found in "Stupid Data Miner Tricks", chapter 6 of the Nerds on Wall Street book, which conclusively demonstrates the utterly meaningless and accidental correlations between butter production in Bangladesh and US stock market returns over a twelve year period. Throw in a couple of other bogus variables, and you get to a regression that "explains" 99% of the market returns over the period, but means nothing.  

Jason did a story in the August 8 WSJ, Data Mining Isn't a Good Bet For Stock-Market Predictions, that revisited the data mine, and included a four minute interview on the poster child for bad quant behavior.

Personal finance columnist Jason Zweig interviews
David Leinweber, author of "Nerds on Wall Street."


The Journal wanted to put in the charts showing the butter numbers along with the S&P, but they have a policy of only using charts for which they have the underlying data, and the 14 year old CD-ROM that contained the Bangladeshi dairy data had gone missing.

Berkeley student Max Dama went in search of the Bangladeshi butter series, to get the WSJ some data in time for the deadline. Alas, some things, like 20 year old butter numbers, are hard to find even on the web*.  In a display of true data miner machismo, Max shifted his sights to mine elsewhere and found a NEW AND IMPROVED highly predictive (but useless) leading indicator for the US stock market.

By data mining we've found that the number of buried treasures discovered in England and Wales is predictive of the Dow Jones Average 1 year later. 92% of the variance, i.e. correlation=.9187, of the DJIA is "explained" by the treasure discoveries.  (In compliance with the Journal's admirable standard, the data and source details are here.)

 

 
Note to World: This is why you should hire Berkeley students.
_______
* Butter in Bangladesh update. The old data is still AWOL, but the UN does cough up this rather nasty butter data. Unlike the original data, there are lots of missing values, and it shows imports and exports, instead of production. FYI, Bangladesh imports about a thousand times more butter than it exports. The investment applications of the Bangadeshi Butter Balance are left for further research.

"Nerds on Wall Street" Book is out now

It's finally hatched. The book, "Nerds on Wall Street: Math, Machines and Wired Markets" by David Leinweber, with a foreword by Ted Aronson, is now published (Wiley 2009).  

It is available at Amazon and all major booksellers.
The new Nerds on Wall Street website is at http://nerdsonwallstreet.com.

NOWS should appeal to both techies and financial types with a sense of humor.

The whole thing can be searched with Amazon "look inside". Some pieces can be had here:

Chapter 1 - An illustrated History of Wired Markets, featuring never-before published photos of livestock on the floor of the Chicago Merc and the FIrst Nerd on Wall Street.

Table of Contents  Index

Videos from related talks are at the bottom of this post.

I'm gratified to see that some distinguished readers had nice things to say about it on the back cover, and inside.

Praise for "Nerds on Wall Street"

“Leinweber isn’t half as crazy as people said! He foresaw the profound change that wired technology would bring to markets (robots trading millions of shares in six milliseconds). Now he nails the Stupid Financial Engineering Tricks that dumped the markets, and offers his patented, sound insights on how the nerds will help bring us back.”

Jane Bryant Quinn,
Financial columnist for Bloomberg.com and Newsweek
Author, “Smart and Simple Financial Strategies for Busy People“

“David Leinweber has been a pioneer in developing and applying advanced technologies in the capital markets. This book is a virtual tour de force survey of many of the key innovations that ‘nerds’ and computational techniques have driven over the past two decades, with key insights on future opportunities in this area. It is a highly engaging, insightful, and entertaining book that will benefit all investors who want to understand the increasingly important role that technology plays in the financial markets.”

Blake Grossman
CEO, Barclay’s Global Investors

“Leinweber leads his readers through a largely unexplored forest, turning over ordinary-looking rocks to reveal hidden colonies of peculiar creatures that feed on moldering mounds of numbers teeming with trailing zeroes. His book is absorbing, instructive, and very, very funny.”

David Shaw
Founder D.E.Shaw & Co.

“Through the lenses of finance “nerds”, Dave Leinweber recounts the quantitative and technological revolution in equity trading. The book is humorously written but it is serious and insightful. It makes an important contribution to our understanding of financial innovation and the evolution of the capital markets.”

André F. Perold
George Gund Professor of Finance and Banking
Harvard Business School

“Finally, a book that rightly honors the pocket-protected, RPN-loving, object-oriented, C-compatible, self-similar Wall Street quant! This is a delightfully entertaining romp across the trading floors and through the research departments of major financial institutions, told by one of the early architects of automated trading and a self-made nerd.”

Andrew W. Lo
Harris & Harris Group Professor
Director, MIT Laboratory for Financial Engineering
MIT Sloan School of Management

“David Leinweber is one of the great financial innovators of our time. David possesses a unique combination of expertise in the fields of money management, artificial intelligence, and computer science.”

Blair Hull
Founder, Hull Trading & Matlock Trading

“An important, accessible, and humorous guide to today’s electronic markets. Like Capital Ideas mixed with Being Digital, as told by Steve Martin.”

Frank Fabozzi
Professor in the Practice of Finance, Yale School of Management
Editor, “Journal of Portfolio Management“

“Most new technologies are exploited first by “alpha geeks,” the folks with the skills to push the envelope. This is as true on Wall Street as it was on the web. David Leinweber was one of those alpha geeks, but is also the first to chronicle the innovation process from early adopter to mainstream acceptance.”

Tim O’Reilly
Founder & CEO O’Reilly Media

“NOWS is a thoughtful, funny and comprehensive history of the overlooked role geeks have played in our financial markets from the earliest days of telegraph, to risk management systems in the current credit crisis. The book is an irreverent “I Was There” chronicle of how our financial markets were formed from silicon, savvy and software. Highly recommended.”

Paul Kedrosky
Infectious Greed
Senior Fellow, Kauffman Foundation
Consulting Strategist, Ten Asset Management

“For decades Dave has not only understood more investment technology than anyone, but with patience and a great sense of humor, he has made the effort to explain it to his less tech savvy friends. Nerds on Wall Street is a home run for us all.”

Richard Rosenblatt
CEO, Rosenblatt Securities

“Clear, light language and wry humor mask David Leinweber’s exhaustive compendium of technological innovations for and impacts on asset trading. Leinweber brings an entrepreneur’s experience and an academic’s perspective to financial technology; and has produced the definitive work, as up-to-date as it is encyclopedic.”

David K. Whitcomb
Founder and Chairman Emeritus, Automated Trading Desk
Professor of Finance Emeritus, Rutgers University

“Nerds on Wall Street is a wild, funny ride though the technological changes that underpin modern financial markets. You will find yourself laughing out loud at what could otherwise be a dry subject. And, if you’re not careful, you might even learn something!”

Richard R. Lindsey
Principal, Callcott Group LLC
Chairman, International Association of Financial Engineers

“If you’re interested in what computers are doing with your money, then this book is for you.”

Richard Peterson MD
Managing Director MarketPsy Capital LLC
Author, “Inside the Investor’s Brain“

“In David’s words, the stock market is a “victim not a cause” of the great mess of 2008. It’s refreshing to read a book with such insight during these difficult times. I applaud David Leinweber for this timely masterpiece.”

Bill Aronin
Co-founder Quantitative Analytics, Inc.
Sr. Manager, Thomson Reuters

“Dr. Leinweber continues to be a patron saint of any nerd who stumbles onto Wall Street. Many of his most insightful ideas are here in this book, the utility of which are only matched by the humor of their presentation. As the markets have changed in 2008, the need to collect, process, and understand novel information sources has never been greater. ”

Jacob Sisk
Infoshock, Yahoo

“Thoughtful insights covering trading, investment practice and system design encased in humor by an expert in all four: a good and practical read.”

Evan Schulman
"Father of Program Trading", Founder, Tykhe, LLC

“David is one of the top practitioners in the fields of textual analysis and sentiment and its application to trading. Leveraging “smart” machines to parse and extract signal from massive quantities of textual data is hard, and David’s work has put him at the vanguard of the next wave of alpha generation.”

Roger Ehrenberg
Information Arbitrage
IA Capital Partners

Video Material

There are talks based on parts of the book available on YouTube

--------

"If you had everything Computationally, where would you put it, Financially?" - Google Tech Talk, about 1 hour long.

--------

"Markets & Technology"  Light 15 minute lunch talk,  with many fine pictures of livestock. Nicely broken into four YouTube size snippets.

http://www.youtube.com/watch?v=R_OdzQrBvGE

1-Introduction

 

http://www.youtube.com/watch?v=OQt_YsoQ6k4&feature=channel

2-markets

 

http://www.youtube.com/watch?v=gLxwxIfXrz4&feature=channel

3-technology

 

http://www.youtube.com/watch?v=fmDB9-7XjOQ

4-computers

Money:Tech

I have been shamed into going back and putting something new on this blog. Attending the O'Reilly Money:Tech conference in NY in Feb 08, and seeing rampant real-time blogging was a wake up call.

An usual conference, with a mix of nerds and Wall Street sorts, and more than a few hybrids such as yours truly. The presentations from Money:Tech are here.

Stupid Data Miner Tricks

This collective web thing actually works. I got an email from ace quant investment manager John Bogle who'd seen a post from  Paul Kedrosky.  Both were looking for a copy of Stupid Data Miner tricks, a paper in the current Journal of Investing.

The JoI is not fully onboard the "information wants to free" train, so as a good citizen of the interweb series of tubes, I'm depositing an earlier version right here. Download dataminejune_2000.pdf

Here's the introduction:

Disraeli's warning that "there are three kinds of lies: lies, damn lies and statistics"  is particularly true when too much computation is applied to too little data. This paper presents some egregious yet instructional examples of data mining, and describes ways to avoid similar mishaps.

It started out as a set of joke slides showing silly spurious correlations over ten years ago. These statistically appealing relationships between the stock market and diary products and third world livestock populations have been cited often, in Business Week, the Wall Street Journal, the book “A Mathematician Looks at the Stock Market”, and many others. Students from Bill Sharpe’s classes at Stanford seem to be familiar with them. This was expanded, to have some actual content about data mining, and reissued as an academic working paper in 2001. Occasional requests for this arrive from distant corners of the world. So I’d like to thank the editors of the Journal of Trading for publishing this.

Without taking a hatchet to the original, the advice here is still valuable, perhaps more so, now that there is so much more data to mine. Monthly data arrives as one data point, once a month. It’s hard to avoid data mining sins if you look twice. Ticks, quotes, and executions arrive in millions per minute, and many of the practices which fail the statistical sniff tests for low frequency data can now be used responsibly. New frontiers in data mining have been opened up by the availability of vast amounts of textual information. Whatever raw material you choose, fooling yourself remains an occupational hazard in quantitative trading.

PS - DIY dataminers will want to check this: http://swivel.com/

 

Algo vs. Algo - With Pictures and Links

Algo vs. Algo is in the Febrauary 2007 issue of Institutional Investor Alpha Magazine
on pages 44-51.

Alas, the financial visualization examples could not be included in the print version. And copying long links is tedious, so I offered an electronic version with links and pictures.
It's downloadable here Download AlgosEdge0103.pdf

Here's a nice sample from Oculus, which will even move if the animated gif below works when you click the picture.

What you see (here or at the Oculus site) is a movie of the microstructure of the market for ORCL.  The limit order book shows sells in red or yellow and buys in blue and green. The heights of the "buildings" show size. The trade history recedes from the front on a visual conveyor belt, with the heights of those purple buildings corresponding to trade size. A striking feature is that when the bulk of buildings on the  buy or sell side of the book show a large imbalance, the price tends to move in that direction.

Hmm, maybe there is something to what they say about supply and demand.

This is suggestive of how a trader steering an algorithm might view the process. There would be overlays indicating which of the trades were yours, and the control parameters and levels for the algo.

Information Driven Price Moves - A Case Study

Download CaseStudy1.pdf

This is a micro-level trace of how information is disseminated and incorporated in financial markets in a era of great flux in news and digital “news-like” information. The case involves Accentia Biopharmaceuticals, which was the largest percentage gainer in US stock markets on the morning of Oct 19, 2006.

Several interesting issues raised here.

• How should  investors and traders, and their information providers, should deal with disintermediated information?      
      
• There is more news than people can handle, and far more distintermediated news. Some means beyond basic keyword search is needed
     
  
• Are   examples like this one, which has language that would be of interest any      time, useful as the basis for “more like this” persistent search?
      
  
• How to obtain the “meta-knowledge” (e.g. in this case, the relationship between      the vaccine name and the company that makes it) needed to interpret news      in a trading and investing context. What combination of automation and      human contributions are appropriate?

 

 

eInformation

     

    Technology has utterly reworked the "plumbing" of markets, dissemination of prices, orders, and clearing. A new frontier in financial technology is dealing with information used to make decisions  that arrives as text or web content, as eInformation

More on eInformation (a slight update of an essay from 2002, by Dave Leinweber and Peter Tufano)

A Transformation in Financial Information

Information is the raw material for any investment or trading strategy, and technology can radically alter the information landscape.  More than a century ago, a new technology - Transatlantic Telegraph Cables - had a major impact on the functioning of financial markets by suddenly bringing the prices of securities between New York and London in line with one another. The Internet may offer a similar revolution in communication and may influence financial markets. Recent years have witnessed the creation of new means by which information, opinions and analyses can be shared among investors. Information frictions are dramatically reduced. Information is impounded in prices much faster. Press releases used to be read by editors, now they are read by everyone. Local newspapers were read locally, now everything is global. Information sharing used to occur within small groups, now there are hundreds of millions of people reading and writing messages on the “Internet wall”. Since the late 90s, we have seen an astonishing growth of freely accessible information on the Net, an equally astonishing growth in the number of people able to act on this information. There is more and more information (and disinformation) available, and with over 20 million online brokerage accounts, it is impounded in prices at ever increasing speeds.  This transformation is dramatic.

The e-Information Project

Hundreds of articles on the informational efficiency of financial markets have been published over the past quarter century in major finance and accounting journals.  However the changes brought about by the net have created new opportunities to learn from empirical examination of the information in markets and the relationship of this information to the price formation process, to returns, to volatility, volume, and liquidity.  Which types of information events lead market response, which types are responses to market activity?

The e-Information group brings together computer scientists and financial economists with the goal of reaching an understanding of the new media for information transmittal and the relationships between market and information. In particular, Internet-based stock message boards provide a real time window into the minds of some individual investors. Our research team, among others around the country, is seeking to understand how this new technologically-enabled form of communication works.  Our research seeks to understand the salience of what we call "e-information": investor sentiment and disagreement, extracted from the Web using statistical language processing.

Below you will find some of our preliminary work, as well links to related commercial and academic work. We are currently working on the construction of a large dataset to test a variety of intriguing propositions about e-information and the stock market. If you find other related links, let us know and we can post them for other researchers to share.

Sanjiv Das, Santa Clara University
David J. Leinweber, California Institute of Technology
Francisco de Asis Martinez-Jerez, Harvard Business School
Peter Tufano, Harvard Business School

Our Research:

• "e-information". Sanjiv Das, Asis Martinez-Jerez, and Peter Tufano.  In this clinical study, we create a new micro-database of much of the information flow about four stocks for a period of six months, including a novel database of e-Information. We define e-Information as estimates of the intensity and dispersion of investor sentiment extracted from four major stock chat message boards. We combine this e-Information with other components of the traditional information set (formal press releases by the firms, filings, analyst revisions, and news stories available electronically) to create an intensive database of as close to the full information flow as is practical. We document certain patterns in the release of information and show that there are suggestions of links between our measure of e-Information and contemporaneous stock returns and implied volatility. While e-Information does not seem to predict subsequent stock price returns, it may be related to the implied volatilities taken from options prices. (Revision dated November 2001)
• "Yahoo! for Amazon: Opinion Extraction From Small Talk on the Web" Sanjiv Das and Mike Chen. This paper describes the algorythm used to extract sentiment from the web. In addition to the paper, additional resources are given below.
• Seminar slides (pdf)
• Sentiment Index Generator
• Example plots of stock prices and sentiment (n.b., stock price is measured in "dollars" and the sentiment is measured in a new metric, called "sents")
• "Three hundred years of stock market manipulation," David Leinweber and Ananth Madhaven, The Journal of Investing 10 (7/Summer 2001), 7-16. This paper shows how stock chat boards can be used to manipulate stock prices.

Commercial sites seeking to understand how e-information works:

• www.monitor110.com
• www.finance.google.com
• www.cyc.com
• www.clearforest.com

Blogs dealing with eInformation:

• www.informationarbitrage.com

Selected related research:

• Antweiler, W. and Murray Frank, 2001, Is all that talk just noise? The information content of internet stock message boards, unpublished manuscript
• "Catching the Buzz," Scientific American, November 2001, 30-32
• M. Hirshey, V. Richardson, & S. Scholz, "How Foolish are Internet Investors?" Financial Analysts Journal, Jan/Feb 2000, 62-69
• Tumarkin, Robert and Robert Whitelaw, 2001, "News or Noise? Internet message board activity and stock prices," Financial Analysts Journal 57, 41-51
• Wysocki, Peter, 1999, Cheap talk on the Web: The determinants of postings on internet stock message boards, unpublished manuscript
• The AEnalyst Project, by Victor Lavrenko, Matt Schmill, Dawn Lawrie and Paul Ogilvie, UMass Computer Science Department.  This group is extracting sentiment from news stories.