I'm a five a month blog galley slave for Forbes, so click here to read them
Are Algorithmic Monsters Threatening The Global Financial System?
Scott Patterson’s new book “Dark Pools” has some remarkable quotes. “We feel we created a monster,” says one of the creators of Island, the pioneering electronic securities exchange.
Bye-Bye, Wall Street: New Flavor Of Big Data May Be More Lucrative For Quants
Back in the old days, this was a famous cartoon about the web.
Now, not only do they know you’re a dog, they know your size, preference in kibble, chew toys, and if you have fleas. It may sound small, but it’s in the Big Data-verse.
In many ways, using the "Big Data" from people's travels on the web is the new Quant Frontier. I've learned a bit more about this from some ex-quants now busy applying bleeding edge methods to new problems.
Are web optimizers the new quants? Their data is bigger.
The Forbes post (“Speed Is Making The Market Dangerous. Nascar Shows How We Can Make It Safer.“) was about the kind of big data produced in financial markets. People are interested in that because of the perceived value in it.
But if we measure by entrepreneurial energy, especially in Silicon Valley, there is more perceived value in the new flavor of Big Data than the old. The Big Data that tracks what people do on the net.
This idea was particularly vividly stated to me by George John, the CEO of Rocket Fuel, a fast-growing Big Data science firm in the Valley. George, and his lead quant jock, Mike Benisch. Both have quant investing cred, having worked on Wall Street, and read “Nerds on Wall Street,” and they invited me over for a visit.
George’s insight is that “Big Data is the New Quant Investing,” and the more I think about it, as a longtime quant investor, the more I think George is right.
Big Data in Financial Markets plus the "Star Wars" of Algo Videos
Annotated Useful Version of Big Data Talks:
Fast Markets, "New News" & Event Trading
The first purpose of this post is to be able to provide something useful to people who have requested slides from talks I've given on structured and unstructured flavors of Big Data in Finance. The slide files themselves were huge due to all the pictures and movies. Plus, the pictures alone didn't really tell the story, and were often used just for a few seconds.
To make something useful, I went back through the slides, and annotated them with comments based on what was said, removing some and adding others with more detail than could be included in the original talks. Download the newly slimmed down 9M version here.*
“Big Data in Financial Markets” is an overly broad topic by a factor of two because markets have two big data revolutions underway at the same time.
The first revolution is related to structured market data, issues around the flash crash, unstable trading systems, and cyber security of markets. That is the subject of the first portion of this talk.
The other big data revolution in markets is in the usual sense used in the tech press – weakly structured information from textual, web, images, social media, governments and commercial sources.
For readers who want to skip to that portion, it starts at page 31.
A common theme in both parts of the talk is how humans can play well with machines. Future skills for "algo jockeys" and "collaborative intelligence amplified research/traders" will emphasize where people can add value to the algorithms and machinery.
The MUST SEE Kevin Slavin video on Algorithms
The other reason for this post is a bonus, to point out this extraodinary TED talk by Kevin Slavin. It does for algos what "Star Wars" did for metal swimwear. It has a broad perpesctive and almost all the examples are financial.
* Nerd Note: How to make annotated talk files:
Getting the comments to print alongside the slides was something of a challenge. From a Mac, you need to move the slides to Windows powerpoint (I used an old XP era version) and did "send to word with notes", which the Mac ppt won't do. That grows to a 400 meg file, that can be whipped down to 9M using Acrobat pro.
Nerds on Wall Street - In Color, and in Chinese
Wired Markets History In Living Color
The book publishing industry hasn't been pushing the bleeding edge of technology for a while. This is changing a bit as everything becomes digital, but it's been a frustrating experience for many readers and authors to be stuck with lo-res black and white images, when the originals were in living color.
I'd like to thank the Museum of American Finance for running a very fine version of chapter one of the NOWS book "An Illustrated History of Wired Markets" in the 100th anniversary issue of their Financial History magazine. To all of you who have the book and said "loved it, except for the nasty little grey pictures", download what it should have looked like here. Download the Color NOWS Chapter 1
NOWS Book in Chinese
If you are OK with the little B&W pictures, but would like to have the book in Chinese, it's available here.
My thanks to the translator, Prof Wang Zhongu at the Harbin Institute of Technology.
The Flash Crash: Just How Weird Was It? See the movie.
Many buy-side traders tell tales of strange market anomalies that they encounter almost daily. But the mother-of-all-market-anomalies has to be the flash crash of May 6, 2010.
Courtesy of Panopticon and Sybase, I have a movie of what may be the weirdest minutes in market history. We all know that the DJIA dropped a kilopoint and came back within minutes starting at 2:42pm.
At an individual stock level, it was even stranger. Accenture (ACN) traded at $40 and at one cent within seconds. Some stocks actually went up. Some saw multiples of normal volume, but went nowhere.
I've used an animated visualization that shows the S&P500 at an individual stock level in a number of recent taks, and by popular demand, I'm putting it up here. Here's a still frame from the beginning.
The narrator, Peter Simpson of Panopticon, explains how the Aleri Complex Event Processing system was used to generate the animation. Each S&P500 srock is represented by a circle. The size shows market capitalization, and the color shows the sector.
Trading activity is seen as the circles move on the chart. Price changes are on the vertical axis. EXC, shown above, has dropped 60% in this frame. Relative volume is shown on the horizontal axis. Note that FIS has seen 500% of average volume, but not dropped at all in price.
For the movie version of these weird few minutes, Download Panopt-Flash-Short-4-3.
For the IMAX version, put your nose up to the screen and click in the same place. At just over a minute, this movie is slightly faster than real-time.
Event Driven Trading and the 'New News' (SF QWAFAFEW Movie Supplement)
Victim of Technology
I'm reading "Glasshouse", a fine cyber SF book by Charles Stross that has far future historians looking back on the mid 20th to 21st centuries as a "dark age", since all records became digital and vanished in a maze of obsolete incompatible technologies.
We don't have to wait for the 25th century to share the love on this. I confidently loaded up a USB key with fancy moving slides and videos in two flavors for a QWAFAFEW* talk in San Francisco. In moving from a mac to the windows laptop, everything went dark except the pdf version of the slides.
Simulation of a News Driven Portfolio
The talk was:
There are two information revolutions underway in trading and investing. Most of the headlines focus on structured quantitative market information at ever higher frequencies. The other technology revolution in trading and investing is driven by qualitative, textual and relationship information. This is important for people who make their living in finance on scales longer than microseconds, even days.
What constitutes “news” is a moving target, as vendors and investors expand and automate collection from primary and proprietary sources, including social media. This is increasingly used for event driven alpha signals. Demonstrations of this using commercial “state of the practice” news systems from Thomson Reuters are a part of this talk. A model sequestered for nearly a year and simulated, with trading costs, on unseen price and news produced an alpha exceeding 11% over its unseen test period, the first three quarters of 2010.
This last point, of putting the model in "cold storage" and then testing on unseen data would be a good standard for all papers of this sort, many of which appear (along with ours in the Winter 2011 issue) of the Journal of Portfolio Management. Alas, application of this rule would require a name change to the Pamphlet of Portfolio Management.
The slides from the talk are here.
The Missing Movies
The movies were for an Event Study Explorer, a visual, interactive "quantextual" model building tool used to find alpha in news and other sources. Without the videos, I broke a few moves pointing to the still slides and pretending to fill in the motion. That worked poorly, so I am posting the four small movie segments on the Event Study Explorer, built in Spotfire, here. It addresses many of the (valid) critiques of event studies. The narration is by my collaborator and coauthor, Jacob Sisk, of Thomson Reuters.
The movies, in four segments, follow the still picture below.
All of this is available for test drives, contact Rich Brown at Thomson Reuters News Analtyics. ([email protected])
---------
*QWAFAFEW is the torturously arrived at name for a group of quants & Wall Street nerds who wanted to have tech talks with beer and snacks. It started in NY in the 80s and has thrived, meeting more or less monthly in nine cities.
Top Books for Quants, plus HFT and Flash Crash Freebies
Quantnet just published their list of best selling books, and it was good to see "Nerds on Wall Street" on there, along with some classics like 'Liar's Poker".
Two authors were there twice, Dan Stefanica and Steve Shreve, who wrote on the Mathematics of Financial Enginneering, and Stochastic Calculus. If I was a bragging kind of guy, I'd say I was in third place, since "How I Became a Quant" is on there along with NOWS, and while HIBQ is credited to the editors, I wrote the first chapter of 24 in the book, so I can say I'm on the Quantnet list 1.04167 times.
I got the first chapter slot because the editor's charming and literate wife thought it had more laughs than the others and might sell a few books. Who knows? I hear Snooki gets $50K to show up and pass out.
Very technical stuff for the most part. Lots on what John O'Brien calls "financial engineering, with a small 'f' & small 'e'", i.e., option and derivative pricing.
Here are a few more volumes to add to the list for a broader set of topics:
Active Portfolio Management: A Quantitative Approach for Producing Superior Returns and Controlling Risk - Grinold & Kahn
An owner's manual for quantitative equity portfolios. A standard reference.
The Econometrics of Financial Markets - Campbell, Lo & MacKinlay
Many of us like to try the new new thing - neural nets, wavelets, genetic programs, machine learning and the rest. I'm certainly part of that group, but it's always a good idea to know the math. Lots of the latest often turns out to be statistical. One neural net researcher says he knows it's statistics, but "I create artificial neural nets" is so much better at cocktail parties.
All About High Frequency Trading - Durbin
High-Frequency Trading: A Practical Guide to Algorithmic Strategies and Trading Systems - Aldridge
HFT is mostly in the "Those that know, don't tell. Those that tell, don't know." phase of its life, so there are no texts like the two above. These two books tell some.
SEC/CFTC Flash Crash Reports
A May report on Preliminary Findings Regarding the Market Events of May 6, 2010
And September's FINDINGS REGARDING THE MARKET EVENTS OF MAY 6, 2010
By far the best values on these lists (free). A detailed and illustrated look at the stangest minutes in market history. Sample below shows Accenture trading at $30 and a penny in the same second. Can't blame Tiger for this money shot.
(SEC/CFTC Preliminary report, page 35)
-------------------
For those with more space on the bookshelf, Quantnet also published a list of 150 or so books on the Goldman Sachs reading list
Thanks to Bernard Donefer, a professor at CUNY’s Baruch College in New York City for sending the list