Chapter 07 – A Little Artificial Intelligence Goes a Long Way on Wall Street

A Little AI Goes a Long Way on Wall Street: Artificial Intelligence and Securities Trading

“If you give someone a program, you will frustrate them for a day; if you teach them how to program, you will frustrate them for a lifetime.”

This is a history and technical overview of one of the earliest artificial intelligence (AI) successes in securities trading. In the Introduction, I described the early experiences in the late 1980s at the MIT Artificial Intelligence Laboratory spin-offs LISP Machines and Inference to apply their tools and techniques on Wall Street. Once we stopped blowing air at the subject and tried doing something useful with real market data, it became obvious that the LISP world and Wall Street were far from compatible.

LISP was (and is) an elegant, mathematically pure approach to computation that made for some remarkable feats of programming. My very first exposure to anything from the AI world came in 1971, when I was a newbie at MIT. Up in the truly strange Technology Square AI Lab machine room, filled with humming PDP-10s programmed to push the boundaries of computer science (and to operate the vending machines in the hall), someone showed me Macsyma, the first symbolic math program, developed by Joel Moses. Computers had been doing math in the sense of calculations from the beginning. ENIAC did ballistics. Big science machines did big numerical science problems from nuclear physics to meteorology.(1) In all cases, what was going on was that the formulas were in the program; then the machine read in all the numerical inputs and ran with it to produce numerical answers.

The difference in Macsyma was that the formula or equation itself was the input, and the machine produced transformations of formulas or solutions to equations in the same symbolic language used in abstract, non-numerical math. It could take derivatives, do integrals, and do fancy matrix manipulations, all in terms of the x’s, y’s, integral signs, d/dx’s, and all the rest. When you asked for the derivative of “x 3 _ x 2 _ x” you got “3 x 2 _ 2 x _ 1” and, unlike all the programs that preceded it, Macsyma didn’t have to know the value of x to do this.

It was absolutely amazing to see. Macsyma utterly blew us away. The median math SAT score for MIT guys hanging in the AI Lab was 800. Everyone thought that, while they had a tough time getting a date and maybe were a little confused on personal hygiene, they were BSDs when it came to doing integrals and derivatives. And here is this machine solving problems in a second that would take any of us a week (likely with a mistake), and solving problems in 10 seconds that we couldn’t touch. It was a humbling experience, and the first time I experienced awe at what clever people could do with computers.

When the Macsyma symbolic math system was first run, it found hundreds of errors in the CRC Handbook tables of integrals and derivatives. The Handbook , at the time, was in its 42nd edition, and on the bookshelf of every working engineer and scientist. Other programs proved theorems, solved logic problems, and played more than passable chess.

But all of this logical magic came with a great deal of baggage. The showstopper was the long pause LISP had to take periodically for “garbage collection” to recover the memory left behind as programs ran. The ability to change large, complex data structures on the fl y allowed LISP to deal with the complexity of problems like symbolic integration, but the need to clean up after those changes created the need for garbage collection.(2)

Wall Street Equity Hedging and Early LISP Trading Systems

When we ran our first, very simple LISP trading systems demonstrations (crossover rules, for the most part) using recorded data for our visitors from Wall Street, we saw their eyes glaze over when, in the middle of the simulated run, the machine would take a break for a few minutes and we would offer more coffee.

My colleague Dale Prouty, a brilliant Caltech Ph.D. physicist whose metabolism seemed to make his own caffeine, and I quickly realized there was no way LISP systems would fit in trading. Similar realizations, in other contexts, contributed to the AI winter, described in Chapter 2 .

Dale had heard that PaineWebber’s equity block desk was looking for proposals for an “intelligent hedging advisory system” for the desk. Ideally, the block traders would “go home flat,” with no net long or short exposure to the market, to sectors, or to other common equity factors. This was not always possible, so the firm had more overnight risk exposure than it wanted. There were many ways to reduce that risk; portfolios of long or short positions in options, futures, and stocks could be constructed to offset the risk on the desk’s book, and unwound as that risk changed. These differed in their effectiveness as a hedge (all those Greek letters dear to the quants) and in the implementation cost of putting them on and taking them off.

Prouty read everything he could find on hedging, shuffled in what he knew about expert systems, and after a competition with some of the bigger names in whiz-bang computer consulting (IBM, Coopers & Lybrand, and Arthur D. Little, as I recall), he walked off with a million – dollar contract to build an Intelligent Hedging Advisory System (IHAS) for PaineWebber. Dale and I had commiserated over our woeful situation of banging the square peg of LISP systems into the round hole of trading. His new contract let us do something about it. IHAS was pretty specialized, and there were only so many block trading desks on Wall Street, but pieces of the solution had much wider applicability. It would fund the development of software components with much broader appeal.

Early Quantitative Finance Systems for Institutional Equity Investors and Institutional Money Managers

We decided to start a new company, Integrated Analytics Corporation (IAC), to use what made sense from the LISP world, but without LISP and LISP machines. Sun Microsystems was emerging as the platform of choice for serious computation. The DOS-based 640K PCs of the time were great for WordStar and Lotus 1-2-3, but not what you would choose to analyze the torrent of data on a market feed in real time.

Our product, which we called MarketMind (later incorporated into QuantEx), was written in the mainstream language C, and ran on Sun hardware. It included only as much AI as the financial user community could deal with, and integrated tightly with their electronic environment. Computational resources were used to make a simple, highly application-specific user interface. This combination of advanced technologies, appropriately applied, resulted in a system used by many of the largest institutional equity investors and money managers in the United States. The system was directly linked to the New York Stock Exchange (NYSE) and other electronic equity execution channels. This was not a prototype or a proposal. This was real and was in wide use for many years, often generating transaction volumes exceeding five million shares per day.

Traders used a special purpose rule-based language to describe a wide variety of market conditions. The system kept up with high-speed incoming market data in real time. Its displays told traders when, where, and how strongly their specified conditions matched the current state of the market. Trading recommendations were formulated based on those specified conditions. Finally, and most importantly, direct electronic execution channels allowed quick action on these recommendations.

Tight integration with both market data and electronic execution channels, combined with an appropriate, accessible user interface and a high level of support contributed to a major AI success story with MarketMind/QuantEx. The transactions flowing through these systems produced more after tax revenue on a busy day than many other AI applications generated over their entire operational lifetimes.

Prehistory of Artificial Intelligence Applications on Wall Street

Summer 1987. AI godfather Marvin Minsky warns American Association for Artificial Intelligence (AAAI) Conference attendees in Seattle that “the AI winter will soon be upon us.” This isn’t news to most of them. Many of the pioneer firms have been pared down to near invisibility. AI stocks have dropped so low that Ferraris are being traded in for Yugos in Palo Alto and Cambridge.

On Wall Street, the expert systems that were last year’s breakthrough of the century are this year’s R & D write-off. LISP(3) machines can be had in lower Manhattan for 10 cents on the dollar. What went wrong? Minsky and many of the others in the AI community had it exactly right: Overblown expectations, awkward user interface …

>>>>>> READ MORE HERE < <<<<<<

All notes for this chapter about artificial intelligence applications, computerized investing, and Wall Street Analytics:

This article originally appeared (coauthored by Yossi Beinart) in the Winter 1996 issue of the Journal of Portfolio Management. It is reprinted in Nerds on Wall Street with permission.

1. One of those numerical meteorology problems led to the discovery of deterministic chaos, the strong dependence of a result on what was presumed to be meaninglessly small differences in the inputs. This was popularized as the so-called butterfly effect, since the seemingly insignificant pressure changes caused by a fl uttering butterfly, well within the limits of error of barometers used to measure them, could result in wildly different simulated future weather and climate outcomes. James Gleick’s book, Chaos (New York: Viking Penguin, 1987), is the place to start for the story of chaos.

2. Garbage collection is just gathering up blocks of memory no longer needed by the program. It is part of most implementations of the LISP language. It is very useful to programmers, who then don’t have to keep track of memory themselves. It is obviously not a good idea to use in a real-time application like trading unless it can be accomplished without stopping.

3. LISP is a computer language particularly suited to manipulation of symbols (as opposed to numbers). It is widely used in the academic AI community. LISP machines are workstations with specialized hardware to run LISP programs efficiently.

4. First described in “Knowledge-Based Systems for Financial Applications,” D. Leinweber, IEEE Expert (Fall 1988).

5. The names “Gensym” and “G2” are subtle LISP jokes. LISP and other AI programming languages sometimes need to create variables, which need names. Gensym is the name of the “generate symbol” function that returns these names. The first time it’s called, you get “G1,” the second time, “G2,” and so on. When Gensym’s founders, including Ed Fredkin of MIT’s AI Lab, needed names for both company and product, they did what LISP programmers do and called them “Gensym” and “G2.” They are found here:

Wall Street Analytics

Easily find all of the
lowest cost no load
mutual funds and ETFs
Available at major ebook stores:

Amazon -- Kindle/MOBI
index funds list investing books

Apple iBookstore -- iPad/EPUB
index mutual funds investment guide

Barnes & Noble -- Nook/EPUB
no load mutual funds investment guide

Smashwords -- EPUB, MOBI, PDF
low cost mutual funds investment guide

no load index mutual funds
by David Leinweber