Part 3 – Artificial Intelligence and Intelligence Amplification

Artificial Intelligence and Intelligence Amplification in Financial Markets

Securities Markets are Machinery Now.

This raises the question of how to best participate in the world’s new wired markets. People who use information technology most effectively will be rewarded.

Artificial intelligence (AI) as an academic discipline began at the famous 1955 Dartmouth conference organized by John McCarthy from Stanford University and Marvin Minsky from MIT. The goal of the AI pioneers was to create a mind, a human in silicon. One key idea was that the brain was a biological computer so all the researchers had to do was figure out what the brain was doing and put it into an actual computer and they’d be done. This was something that people thought in the 1950s might take 10 years to accomplish, maybe 15 with long lunches.

So far, it hasn’t exactly worked out. In fiction we have the example of HAL, from 2001: A Space Odyssey. Letting the computer do the thinking turned out badly in that case. HAL discovered the lie in the first reel, and quickly moved on to become a paranoid serial killer.(1)

In financial circles there was a lot of sort of irrational technology exuberance as well. As seen in Chapter 2, there were some inspired magazine covers from the magazine Wall Street Computer Review(2) that showed the unrealistic expectations for AI. One, from 1987, depicts Socrates on the steps of the stock exchange surrounded by a horde of PCs, and touts: “Teaching Computers to Emulate Great Thinkers.”

Computers “thinking” like a person didn’t really work out as well as people had hoped. In the academic AI world, the artificial sentients were persistent no-shows. Artificial intelligence got kind of a bad rep on Wall Street as well. To get into the game, you had to buy a LISP (list processing) machine, a $100,000 machine that ran the elegant but arcane AI language of choice, LISP, and related expert system tools with names like Automated Reasoning Tool (ART) and Knowledge Engineering Environment (KEE).

You could prove a theorem or two with this stuff right out of the box, but getting a price for IBM on the screen proved to be a prodigious amount of work.

As recounted in the Introduction to this book, I worked for one of the companies that made these machines—there were two, LISP Machines and Symbolics—and came to the realization that building from the high-concept top down was a poor idea. Here is a picture I used back then to show the user reaction to AI investment systems, showing a mix of fear and rage.

Chapter 7 – A Little AI Goes a Long Way on Wall Street, describes a successful application, built in the early 1990’s, which was much more modest in scope than bringing back the great thinkers. The goal was to get computers to be decent users of other computer systems, which were overloading their users. Recently, this idea has been called intelligence amplification (IA). The systems here were market data and execution systems, and the combination was an early version of algorithmic trading.

The AI/IA-flavored approach to algo trading and market surveillance used in QuantEx and MarketMind, described in Chapter 7, was very effective in many contexts—equity strategies, quant option trading, and monitoring liquidity— but, like spreadsheets, they were clever ways to get the computer to do what you said, and didn’t have anything in the box that would let the machine learn what you should have said and adjust its behavior or models to better meet your needs.

One of the early enthusiasts for these technologies was Henry Lichstein, an MIT graduate then serving as technology adviser to John Reed, chairman of Citicorp. Henry was also on the board of the Santa Fe Institute (SFI),(3) a new high-powered research institute interested in complex interactions of complex entities. SFIdid a great deal of computer experimentation in artificial life (ALife).

The ALife people had much more modest goals than AI; all they wanted to do was build software entities that displayed lifelike behavior. And their efforts were met with early success. A bunch of random ALife birds might fl y in random directions, and not act much like birds. But giving them a few simple rules, like “fl y toward the closest bird,” “go with the flow,” and “don’t hit other birds,” could give rise to distinctly birdlike behavior.

The original ALife flock is Craig Reynolds’ “Boids,” done at Symbolics in 1986.(4) Simulated herding and fl ocking turn out to be of some commercial interest. Those massive stampedes in Disney cartoons, with thousands of , and all those schools of talking fish are descendants of the Boids. Hollywood showed its appreciation by giving Reynolds an Academy Award in 1998.

Like our early algos, the SFIartificial life was dumb, just obeying a few simple handwritten rules, without any ability to learn from mistakes. But the ALife researchers had a good answer—evolving intelligent behavior by mimicking natural evolution. Represent the programs as digital chromosomes, and simulate crossover and mutation, to breed better programs. The original version was called the genetic algorithm (GA), which later evolved into evolutionary computation (EC). Henry Lichstein suggested this approach for use in trading around 1990. The results for ALife had been striking. All sorts of tasks could be learned by the programs operating in an electronic world, which sure sounded like quant investment and trading. ALife programs evolved behavior to solve problems, avoid obstacles, find rewards, and cooperate. These are remarkable to watch.(5)

Chapter 8 – Perils and Promise of Evolutionary Computation on Wall Street, is one of the few detailed descriptions of the use of the GA and EC in finance. Actual chromosomes and fitness functions are included. If that last sentence made your eyes glaze over, skip to Chapter 9. It was satisfying research, and the power of evolutionary search truly teaches you to be careful what you ask for. Evolutionary approaches could be given rules to avoid many sins of data mining, and had many appealing features. As much as I would like to see this work, we don’t see a torrent of EC scientists fl owing toward greater Wall Street. We don’t even see a trickle. Olsen & Associates was a Zurich-based currency trading firm, founded in 1992, heavy with physicists from the Eidgenössische Technische Hochschule (ETH), Einstein’s alma mater. They were keen on genetically adaptive strategies and well funded, but vanished, and few of the principals are still keen on genetic algorithms.

After sending the GA to the back of the breakthrough line in the previous chapter, in Chapter 9 – The Text Frontier, we get to using IA, natural language processing, and Web technologies to extract and make sense of qualitative written information from news and a variety of disintermediated sources.

In Chapter 6 – Stupid Data Miner Tricks, we saw how you could fool yourself with data. When you collect data that people have put on the Web, they can try to fool you as well.

Chapter 10 – Collective Intelligence, Social Media, and Web Market Monitors and Chapter 11 – Three Hundred Years of Stock Market Manipulations, include some remarkable and egregious examples.

>>>>>> READ MORE HERE < <<<<<<

Notes for this section about Financial Artificial Intelligence, Intelligence Amplification, Genetic Algorithms, and Evolutionary Computation in Securities Markets:

1) Sci-fi buffs have a rich amount of material to consider in this context. Vernor Vinge, a computer scientist who has also won five Hugo awards, deals with the topic in much of his work, including his latest novel, Rainbow’s End. If we construct an artificial super-intelligent entity, what will it think of us?

2) Source: Wall Street Computer Review, June 1987. Since renamed Wall Street & Technology, and a useful resource for nerds on Wall Street at www.wallstreetandtech.com.

3) “The Santa Fe Institute is devoted to creating a new kind of scientific research community, one emphasizing multidisciplinary collaboration in pursuit of understanding the common themes that arise in natural, artificial, and social systems.” It was founded in 1984 (www.santafe.edu/).

4) … and still flocking after all these years at www.red3d.com/cwr/boids.

5) Until they start barcoding these into books, printed URLs are annoying, but it is actually worth typing these in. Karl Sim’s MIT video is here: www.youtube.com/watch?v=F0OHycypSG8.

Wall Street Analytics

• Part 4 – Nerds Gone Wild – Wired Markets in Distress (Financial Nerds Gone Wild - Global Markets in Distress The original plan for this book stopped after the three parts that you’ve just read. These parts are about how markets became machines, and about using more machines to pick stocks and trade them electronically, bringing in an assortment of nifty ideas from finance and computer science [...])
• Chapter 12 – Shooting the Moon – Stupid Financial Technology Tricks (Stupid Financial Technology Tricks and Global Economic Collapse Like many others from the stock side of greater Wall Street, I felt blindsided by the events of 2008. “Blindsided” is actually a gross understatement; I felt like the guy who comes home and finds the neighbors were running a meth lab that has exploded and flattened the [...])
• Praise for “Nerds On Wall Street” ("Leinweber isn't half as crazy as people said! He foresaw the profound change that wired technology would bring to markets (robots trading millions of shares in six milliseconds). Now he nails the Stupid Financial Engineering Tricks that dumped the markets, and offers his patented, sound insights on how the nerds will help bring us back." [...])
• Chapter 08 – Perils and Promise of Evolutionary Computation on Wall Street (Using Genetic Algorithms, Optimization Models, and Evolutionary Computation on Wall Street “Be careful what you ask for — you might get it.” My enthusiasm for machine learning, described at the end of the previous chapter, led me to kiss many artificial intelligence ( AI ) frogs. This included many flavors of inductive and explanation - based learning, [...])
• Chapter 07 – A Little Artificial Intelligence Goes a Long Way on Wall Street (A Little AI Goes a Long Way on Wall Street: Artificial Intelligence and Securities Trading “If you give someone a program, you will frustrate them for a day; if you teach them how to program, you will frustrate them for a lifetime.” This is a history and technical overview of one of the earliest artificial intelligence (AI) [...])
• Overview of “Nerds on Wall Street” (Technology has transformed global markets, but this is nothing new. Markets have been shaped by machinery for hundreds of years, and this continues at a rapid pace today. Author David Leinweber—a computer scientist who accidentally stumbled upon Wall Street and became an innovator in the application of modern information technology in trading and investing—is a well-qualified [...])
• Chapter 10 – Collective Intelligence, Social Media, and Web Market Monitors (Web Market Monitors and the Impact of Social Media on Financial Markets "The words of the prophets are written on the subway walls." — Simon & Garfunkel, The Sound of Silence Opinions vary widely on the value of collective wisdom, with ample supporting evidence both for and against. The Internet has many positive examples: The collective ratings [...])
• Chapter 02 – Greatest Hits of Computation in Finance (Computational Finance, Stock Market Analysis, and Investment Trading "A computer does not substitute for judgment any more than a pencil substitutes for literacy. But writing without a pencil is no particular advantage." - Robert McNamara The Journal of Portfolio Management (JPM*) is one of the more upscale investment management publications around. For $500 a year, you get [...])
• Chapter 03 – Algorithm Wars (Algorithmic Trading Strategies and Automated Stock Trading “How about a nice game of chess?” — WOPR computer in "War Games" There used to be two market structures for U.S. equity traders to contend with: the NYSE (for listed stocks) and NASDAQ. Recent counts put the number at roughly 40. Many are sources of dark liquidity, which sounds [...])
• Forward by Ted Aronson (Nerds on Wall Street Forward by Ted Aronson Quantitative finance is not a topic usually associated with laughter. That is about to change with the publication of Nerds on Wall Street. I was first exposed to Dave Leinweber’s wit when he delivered a speech entitled “Nerds on Wall Street.” I believe the event happened 20 or 25 [...])

• Alpha as Life (Passive Investing - Active Investing - Alpha Returns Index funds are passive investments; their goal is to deliver a return that matches a benchmark index. The Old Testament of indexing is Burton Malkiel’s classic A Random Walk Down Wall Street, first published in 1973 by W.W. Norton and now in its ninth edition. For typical individual [...])
• Wired Markets ( Financial Markets - Electronic Markets Not too long ago, going to a stock market meant you would meet lots of new people who were energetically shouting, running around, and making a mess with great quantities of paper. No more. Visiting a financial market now is more like visiting a telephone exchange. It can be a wild ride versus parking your cash in a few money market funds. Computers and network gear [...])
• Nerds Gone Wild – Wired Markets in Distress (Financial Nerds Gone Wild - Global Markets in Distress The original plan for this book stopped after the three parts that you’ve just read. These parts are about how markets became machines, and about using more machines to pick stocks and trade them electronically, bringing in an assortment of nifty ideas from finance and computer science [...])
• A Little Artificial Intelligence Goes a Long Way on Wall Street (A Little AI Goes a Long Way on Wall Street: Artificial Intelligence and Securities Trading “If you give someone a program, you will frustrate them for a day; if you teach them how to program, you will frustrate them for a lifetime.” This is a history and technical overview of one of the earliest artificial intelligence re (AI), and is a far cry from simple financial planning software [...])
• Collective Intelligence, Social Media, and Web Market Monitors (Web Market Monitors and the Impact of Social Media on Financial Markets "The words of the prophets are written on the subway walls." — Simon & Garfunkel, The Sound of Silence Opinions vary widely on the value of collective wisdom, with ample supporting evidence both for and against. The Internet has many positive examples: The collective ratings [...])
• Artificial Intelligence and Intelligence Amplification (Artificial Intelligence and Intelligence Amplification in Financial Markets Securities Markets are Machinery Now. This raises the question of how to best participate in the world’s new wired markets, and this is anything but simple. People who use information technology most effectively will be rewarded. Artificial intelligence (AI) as an academic discipline began at the famous 1955 Dartmouth conference organized by John McCarthy from Stanford [...])
• AI, IA, and the New Research (Hunting Investment Alpha and Trading Alpha from Online News, Social Media, and Rumors Alpha hunters are always looking for new territory. When a strategy becomes known and used by too many players, the collective market impact of getting in and getting out will squeeze out all the profit juice, and only the lowest-cost transactors (large sell-side [...])
• Stupid Data Miner Tricks (To Err Is Human. To Really Screw Up, You Need a Computer. — Popular Campus T-shirt, circa 1980 Stupid Data Miner Tricks in Quantitative Finance This chapter started out over 10 years ago as a set of joke slides showing silly, spurious correlations. Originally, my quantitative equity research group planned on deliberately abusing the genetic algorithm (see Chapter [...])
• Greatest Hits of Computation in Finance (Computational Finance, Stock Market Analysis, and Investment Trading "A computer does not substitute for judgment any more than a pencil substitutes for literacy. But writing without a pencil is no particular advantage." - Robert McNamara The Journal of Portfolio Management (JPM*) is one of the more upscale investment management and financial article publications around. For $500 a year, you get [...])
• An Illustrated History of Wired Markets (An Illustrated History of Wired Capital Markets "Progress might have been all right once, but it has gone on too long." -- Ogden Nash This chapter is based on a number of ever-evolving dinner and lunch talks I have given over many years, all called “Nerds on Wall Street" irrespective of their actual subject. Many financial conference [...])
• A Gentle Introduction to Computerized Investing (Computerized Investing, Index Funds, Quantitative Investing, and Active Management “Life would be so much easier if we only had the source code.” — Hacker proverb The beginning of index investing in the 1970s was the result of a convergence of events, one of those ripe apple moments. Institutional investors began to use firms like A.G. Becker to actually [...])
• Three Hundred Years of Stock Market Manipulations (300 Years of Stock Market Manipulations - From the Coffeehouse to the World Wide Web's Stock Manipulations In previous chapters, we saw that many of the changes in securities markets brought about by information technology in general and the Internet in particular are positive, democratizing access to markets and information. We also saw that technology is [...])