Algorithmic trading

[Storm]

Has anyone here ever dabbled in algorithmic trading?  I'm fascinated by it.  I do have something of a background in programming, being a computer science major, and being a system engineer/administrator in my job I do write a lot of scripts.

Many hedge funds and quant funds take this approach to trading.  When you go to visit the trading floor, you find that most of the traders are writing small perl scripts that take an idea they had and execute it automatically.

There is also a forum at elitetrader.com where many of these algorithmic traders hang out.  Basically, in order to do this, you need:

• A home computer - almost any computer will do as some people simply code in Microsoft Excel macros.
• A reliable and fast Internet connection - you need your trades executed quickly.
• An Interactive Brokers account - they will send you accurate tick by tick market data feeds.

You can download historical market data feeds from IB as well.  The golden rule is that if you think you have a money making algorithm, you back test it on historical data first before ever pushing your algorithm live to trade with real money.

I also have a book on this, Professional Stock Trading - System Design and Automation, which has actual algorithms and code in it (the author claims to have been profitable with some of them, but by now they may be so outdated that who knows).

http://www.amazon.com/Professional-Stoc ... 627&sr=8-1

The book has excellent reviews, although I haven't read much of it yet.  I'm curious to know if anyone on this board has ever tried this approach to trading, in a VP of course.

[akratic]

My day job is in algorithmic trading.  Two thoughts:

1) Finding profitable algorithmic trades is *very hard*.  We have a team of people with MIT degrees, etc., and it's rare for us to stumble upon something that works.  Good algorithmic trading ideas take teams of smart people years to develop.

2) If you do find a profitable algorithmic trade, you'd have to be crazy to fund it yourself.  Suppose you had an idea that would return 100% on up to $10 million with low risk.  Which would be better?
a) invest your own $300k, turn it into $600k minus colocation costs, infrastructure costs, exchange fees, clearing fees, etc.
b) invest $10 million of someone elses money, turn it into $20 million, keep a 30% cut and walk away with $3 million.

For b) you might think it would be hard to find someone to give you $10 million to trade, but the shortage is in people who can come up with profitable algorithmic trades, not in people/firms with $10 million who would be interested in easy money.

Algorithmic trading is a fun challenge, but unlikely to be something you could conquer as a side hobby.

[Storm]

Thanks for the insight, akratic.  I came to pretty much the same conclusion, that it would definitely take a full time approach to learning it for many years before you could beat the average returns of the market.  I also decided that as a hobby, I would probably never be profitable at it, which is why I haven't done it with my own money.

If I am able to make early retirement, perhaps this would be a fun new career to start.  If you had to recommend a course of study or books to read to begin learning, what would you recommend?

[Lone Wolf]

Many hedge funds and quant funds take this approach to trading.  When you go to visit the trading floor, you find that most of the traders are writing small perl scripts that take an idea they had and execute it automatically.

I'd always have this fear that I'd use the wrong character somewhere in my Perl code and wipe out my life savings.  Perl code always kind of looks like a long string of cartoon swearing to me!

Check out this great submission from the old Obfuscated Perl Contest:

Code: Select all

@P=split//,".URRUU\c8R";@d=split//,"\nrekcah xinU / lreP rehtona tsuJ";sub p{
@p{"r$p","u$p"}=(P,P);pipe"r$p","u$p";++$p;($q*=2)+=$f=!fork;map{$P=$P[$f^ord
($p{$_})&6];$p{$_}=/ ^$P/ix?$P:close$_}keys%p}p;p;p;p;p;map{$p{$_}=~/^[P.]/&&
close$_}%p;wait until$?;map{/^r/&&<$_>}%p;$_=$d[$q];sleep rand(2)if/\S/;print

Crazy stuff.

[stone]

It sounds facinating and fun to me but it also seems to me to be theft. You are getting stuff by being more sneaky than everyone else. Am I totally mixed up in thinking like that?

[Storm]

I'd always have this fear that I'd use the wrong character somewhere in my Perl code and wipe out my life savings.  Perl code always kind of looks like a long string of cartoon swearing to me!

Check out this great submission from the old Obfuscated Perl Contest:

Code: Select all

@P=split//,".URRUU\c8R";@d=split//,"\nrekcah xinU / lreP rehtona tsuJ";sub p{
@p{"r$p","u$p"}=(P,P);pipe"r$p","u$p";++$p;($q*=2)+=$f=!fork;map{$P=$P[$f^ord
($p{$_})&6];$p{$_}=/ ^$P/ix?$P:close$_}keys%p}p;p;p;p;p;map{$p{$_}=~/^[P.]/&&
close$_}%p;wait until$?;map{/^r/&&<$_>}%p;$_=$d[$q];sleep rand(2)if/\S/;print

Crazy stuff.

Haha.  I get the same feeling and I write Perl a lot.  Sometimes I can't read code I wrote if I read it like a year later - I have to use a lot of comments just to make sure.

I think the code above is printing out "Just another Perl / Unix hacker" but it's hard to tell for sure.

[Lone Wolf]

Haha.  I get the same feeling and I write Perl a lot.  Sometimes I can't read code I wrote if I read it like a year later - I have to use a lot of comments just to make sure.

I think the code above is printing out "Just another Perl / Unix hacker" but it's hard to tell for sure.

I hear that!

Spoiler alert: Here's the solution on how that thing works.  It involves multiple simultaneous processes, interprocess communication, and a whoooooole lotta weirdness!

[akratic]

If you had to recommend a course of study or books to read to begin learning, what would you recommend?

I asked the same question when I started, and the answer I got was that this stuff isn't in books.  Now that I know some stuff about algorithmic trading, I would have to agree.  There's some overlap with gambling stuff like expected value and the Kelly criteria.  There's some pricing foundation stuff you can get from options books and treasury yield curve books, etc.  Software engineering books can be relevant because often the code required is quite complex.  But at the end of the day I don't think you can directly learn much of anything about real algorithmic trading from the books and courses currently available.  

It sounds facinating and fun to me but it also seems to me to be theft. You are getting stuff by being more sneaky than everyone else. Am I totally mixed up in thinking like that?

Yes.  But you have to know a fair bit about algorithmic trading to understand why.

Most people who are relatively uninformed about algorithmic trading think that it's taking tiny amounts in many small transactions.  The truth is it's more often *giving back* tiny amounts in many small transactions.

I enterred the business as a skeptic, taking the word of my good friend who I would work with.  But I've come to see that if I can pull the idea I'm working on off, it would indirectly benefit millions of people.  The subtle and indirect benefit thing bothers me sometimes.  My girlfriend is a teacher and the good she does is so easy to see and hear from her students.  

On the other hand, what my idea will do if it works is make millions of transactions slightly cheaper for all parties involved.  Due to ripple effects, this could help the world out quite a lot in a way that no one will ever notice.

You can think I'm deluding myself about this if you want.  I'm over trying to convince people who don't know much of anything about how algorithmic trading actually works, but have strong opinions about it anyway.  (I'm not saying that you're doing this, but it's very common.)

Anyway, no, I don't think algorithmic trading is sneaky or theft or immoral or taking or wrong.

[Storm]

It sounds facinating and fun to me but it also seems to me to be theft. You are getting stuff by being more sneaky than everyone else. Am I totally mixed up in thinking like that?

Stone, a lot of algorithmic trading is "pair trading" which might be very familiar to PP adherents.  You enter two positions simultaneously with the idea that one is overvalued and one is undervalued.

From "Professional Stock Trading":

Figure 2.4 is an example of a Short A-Long B entry.  Stock A (ATVI) has become overvalued relative to Stock B (THQI) because the Spread has risen above the upper Spread Band.  As soon as the Spread crosses back below the upper SB, we short 1400 shares of ATVI at a price of 26.70 at 1:40 pm and buy 600 shares of THQI at 54.10. 

The Spread crosses below zero right at 9:35 am the next day, at which point the ATVI short trade is covered at 26.07 for a profit of 0.63 points.  Simultaneously, we sell THQI at 55.96 for a profit of 1.86 points on the long side of the pair.  Together, the pair trade nets a total profit of $1,998, not including slippage and commissions.

I'm not seeing how this is anything like theft.  I think there is a misperception that algorithmic trading is all High Frequency Trading.  That is not the case.  It could just be an algorithm looking at a particular sector where you've noticed that "when the price of oil is above X and this stock A has a lower P/E than stock B, go long A and short B."

[Storm]

I asked the same question when I started, and the answer I got was that this stuff isn't in books.  Now that I know some stuff about algorithmic trading, I would have to agree.  There's some overlap with gambling stuff like expected value and the Kelly criteria.  There's some pricing foundation stuff you can get from options books and treasury yield curve books, etc.  Software engineering books can be relevant because often the code required is quite complex.  But at the end of the day I don't think you can directly learn much of anything about real algorithmic trading from the books and courses currently available.  

Is the book I have, Professional Stock Trading, any good?  There is also a great book that I plan on reading called "Trading and Exchanges" by Larry Harris.  It's more of a general "if you want to know how the stock market really works" book though.

[MediumTex]

akratic,

What draws you to the PP?

I tend to think that PP people are those who believe that it is hard or impossible to reliably pull money out of the market by using any kind of trading system, while algorithmic trading seems to be premised upon the opposite assumption.

I'm glad you are here, but what attracted you in the first place?

[MachineGhost]

I trade quantitatively using algorithms, but that is not the same as "algorithmic trading" as known on Wall Street.  There seems be a mystique about such trading in general when it is rather old school in terms of origins (late 80's to early 90's).  Any requirement for backtesting ideas and strategies will require them to be put into some kind language that a backtesting engine can read, unless an idiot-proof overlay interface is provided (always with severe limitations).  Nothing more, nothing less.  Likewise for automation.

Unfortunately, the hucksters have since caught on to "automated trading" and the end result is a seemingly never-ending bombastic parade of Expert Advisors being sold for MetaTrader 4 in the currency market.  It certainly doesn't take a rocket scientist to code up something that is curve fit to a limited time in past history, and then sell it using a chart of the hypothetical profits.  Any kind of trading must exploit a structural edge about the marketplace to succeed long-term, not data mine a pattern out of randomness.  No exceptions.

IB doesn't provide a true tick-by-tick feed, only snapshots to save on bandwidth.  Nor does talk of tick-by-tick imply "algorithmic trading" is exclusively the domain of intraday.  Even the PP is a "trading algorithm"!  Nor is "algorithmic trading" the same as "high frequency trading" nor is it the same as "automated trading", though all are "quantitative trading".  Technically speaking, "algorithmic trading" refers to "massaging order flow" for lack of a better term.  Compared to the technical analysis jungle, it is the ivory tower of quantitative trading.  Don't get blindsided by all the buzzwords.

For a little perspective, after the death of open outcry, those former floor traders that made the successful transition "upstairs" to "sit at a desk" with a PC are the "High Frequency Traders".  They are providing the liquidity that they formerly did on the floor.  (It is way more exotic/complex than that now, but the end result is effectively the same.)

MG

Has anyone here ever dabbled in algorithmic trading?  I'm fascinated by it.  I do have something of a background in programming, being a computer science major, and being a system engineer/administrator in my job I do write a lot of scripts.

Many hedge funds and quant funds take this approach to trading.  When you go to visit the trading floor, you find that most of the traders are writing small perl scripts that take an idea they had and execute it automatically.

There is also a forum at elitetrader.com where many of these algorithmic traders hang out.  Basically, in order to do this, you need:

• A home computer - almost any computer will do as some people simply code in Microsoft Excel macros.
• A reliable and fast Internet connection - you need your trades executed quickly.
• An Interactive Brokers account - they will send you accurate tick by tick market data feeds.

You can download historical market data feeds from IB as well.  The golden rule is that if you think you have a money making algorithm, you back test it on historical data first before ever pushing your algorithm live to trade with real money.

I also have a book on this, Professional Stock Trading - System Design and Automation, which has actual algorithms and code in it (the author claims to have been profitable with some of them, but by now they may be so outdated that who knows).

http://www.amazon.com/Professional-Stoc ... 627&sr=8-1

The book has excellent reviews, although I haven't read much of it yet.  I'm curious to know if anyone on this board has ever tried this approach to trading, in a VP of course.

[Storm]

IB doesn't provide a true tick-by-tick feed, only snapshots to save on bandwidth.  Nor does talk of tick-by-tick imply "algorithmic trading" is exclusively the domain of intraday.  Even the PP is a "trading algorithm"!  Nor is "algorithmic trading" the same as "high frequency trading" nor is it the same as "automated trading", though all are "quantitative trading".  Technically speaking, "algorithmic trading" refers to "massaging order flow" for lack of a better term.  Compared to the technical analysis jungle, it is the ivory tower of quantitative trading.  Don't get blindsided by all the buzzwords.

Interesting.  When you say "massaging order flow," what exactly do you mean?  Do you mean simply noticing that the bid/ask is a little bit more favorable at Exchange X than Exchange Y so you route your order to X instead?  I assume you're not talking about HFT who seems to want to exploit those penny differences between X and Y by doing a simultaneous buy/sell to both?  As you can probably tell by my questions I really don't know what I'm talking about, but I'm eager to learn anything you have the time to share.

Thanks!

[stone]

akratic, it is good to hear that my anxieties about this seem unfounded. I'm totally ignorant about all of this and it is great to hear from you guys what it is actually about. I agree with machine ghost that the PP is a crude "trading  strategy". From what I can see as a "trading strategy", the PP does benefit from the mistakes of others and much of the gains we get as PP holders come from the poor souls who buy high and sell low. But the effect on the market of people following the PP, is to smooth out such ineptitude induced volatility. By contrast, the "decision moose" and even the "leveraged etf PP" strategies not only harvest money from such volatility, they also feed into amplifying it. For those strategies, the gains do look to me to be gained by something fairly dodgy. You are gaining from other people and at the same time also causing the market to be more mispriced. Is it not true that a lot of trading activity is just such volatility amplifying momentum trading?

HFT activity such as quote stuffing so as to trade ahead of the tape or whatever is pure theft though isn't it?
Even front running funds as they try and track an index seems to me to be parasitic. I suppose the main thing is that the market does not get structured in such a way to try and maximise the ability of such parasitism. To me regulatory capture by parasites is the real danger.

As I said, I'm glad of being put right about any of this.

[akratic]

akratic,

What draws you to the PP?

I tend to think that PP people are those who believe that it is hard or impossible to reliably pull money out of the market by using any kind of trading system, while algorithmic trading seems to be premised upon the opposite assumption.

I'm glad you are here, but what attracted you in the first place?

I've given a lot of thought to this question, so prepare to have your ears talked off.

As Harry Browne said, you build your wealth through your career, and you protect it with the Permanent Portfolio.  Algorithmic trading is the best carreer for me, and the Permanent Portfolio is the best place for my personal investments.  I think there are three relevant parts to this discussion: 1) market efficiency, 2) what algorithmic trading and the PP have in common, and 3) why algorithmic trading is the best career for me.

1) On the timescale and transactions costs of my personal investments, I believe the market is efficient, and furthermore, it's efficient mostly because of the hard work of a bunch of algo traders.  For an example, consider the treasury yield curve of 2Y, 5Y, 7Y, 10Y and 30Y bonds.  For my personal investments, I do not wonder if I can get a slightly better deal on 10Y bonds compared to the rest.  I can't.  These bonds have mathematical and non-mathematical relationships, but I don't have to worry about it, because the algo traders are keeping it all efficient.  (The real amazing thing about the way they do this is they do it not by taking but by giving...)  Anyway, because the algo traders made the market efficient, I just pick the duration bond I want.  30Y.  Bam.

Consider what would happen if some huge entity bought a ridiculous amount of 5Y bonds.  This effect should ripple throughout the yield curve.  It's the algos that handle this, almost instantly repricing 2Y, 7Y, 10Y, and 30Y bonds.  It's also the algos that sold most of the 5Y bonds to the big entity.  They did all this on the scale of milliseconds, and for my personal investments, I don't have to think about any of it.  All I have to care about for my personal investments is what duration I want.  There's inefficiency that can be captured on the millisecond time scale of the algorithm, but not on the time scale of my Vanguard Brokerage.

It's the same deal with currencies, and the same deal with equities, and the same deal with energies... but I think bonds are the simplest place to see how a bunch of hardworking MIT grads can contribute to the market.

2) There is a lot in common between the PP philosophy and the philosophy behind the type of algorithmic trading that I work on (there are many).  Two big overlaps are accepting that the future is unknown, and focusing on hedging and risk management.

Imagine you have no idea what direction bonds are going to move, but you understand the relationships between the various duration bonds better than anyone in the world.  You can profit whether the market goes up or down if you trade these relationships.  Trying to predict what direction an individual bond is going to move in the future is basically impossible.  Trying to predict what exactly should happen to 2Y, 7Y, 10Y and 30Y bonds following a big 5Y move is tenable, provided you and your team are very smart.  Anyway, just like the PP, many algo trades don't purport to be able to divine the future.

In algo trading, half the battle is valuation/pricing, and the other half of the battle is risk management and hedging.  Most personal asset allocations are basically blind to risk management, and are essentially completely unhedged bets for prosperity, for example.  The PP is the only asset allocation I know of that takes seriously the stuff I grapple with every day: controlling risk, considering the trade off between volatility and expected return, financial products that can offset each other, etc.

Given how much the PP and algo trading have in common, the real thing I can't figure out is why more algorithmic traders don't have their personal money in the PP.

3) I'm going to have to get personal to talk about why algorithmic trading is the best career for me, but this is a forum, so I suppose that is appropriate.  I have three MIT degrees (philosophy, computer science, graduate level computer science).  I am ambitious, and I know a lot of other ambitious people.  (I'm not materialistic though, or into status, or flaunting wealth, or fame, or any of the things that often get associated with ambition.)  My pure strength is software engineering, but I'm good with numbers and games too.

After much introspection and observation of my peers, I've decided there's five paths for ambitious software engineers:
1) corporate
2) startup
3) drop out
4) freelance
5) finance

The problem with corporate is it's too easy, and there's a big disconnect between production and compensation.  The difference between software engineer production can be orders of magnitude.  You'll have to take my word for this if you've never experienced it yourself, but basically some projects that would take an unskilled software engineer two months can be done by very good software engineers in two days.  The corporate world fundamentally doesn't get this, and assumes all engineers produce about the same, and indeed pays every engineer about the same, which can you verify on salary.com.

The problem with startups is you need luck and business skills.  You either have to attach yourself to a business person or early stage startup, praying that you get lucky if they turn out to be awesome at the business stuff.  Or you have to wear the business hat yourself, which many engineers are spectacularly bad at.  Startups require all these skills like public relations, networking, design, marketing, support, willingness to deal with paperwork and bureaucracy, interacting with people, etc., that are not often found in my peer group.  Many of my friends could have single handedly done the initial engineering for facebook, twitter, or many of the recent successful startups... provided they were unfathomably lucky enough to team up with the right business guy at the right time.

The problem with dropping out is a) you need some money in life and b) almost by definition it's hard for ambitious people to do nothing.  You might be able to drop out with $300k or $500k saved up (see ERE) and certainly with a few million, but until then you need a career.  Note that the corporate path is kind of a dropping out hybrid, as you can coast at corporate software engineering jobs doing only a few hours a day of real work, and then spending the rest of your time dicking around on the internet, talking with coworker-friends, and thinking about personal projects.  You're still trapped at your desk though, and answering to a boss and alarm clock.

The problem with consulting/freelancing is it's fairly hard work for the return.  You get paid by the hour usually, and even if it's a high hourly rate, the compensation can never touch the compensation from a successful startup or finance.  In addition, you're stuck needing many of the skills you'd need to do a startup, forced to spend time on sales, negotiating, processing invoices, answering phone calls, etc.  If you really want a high rate per hour of *actual effort*, corporate slacking is probably better than freelancing.

The problem with finance is that uninformed people totally hate it, and often your coworkers suck.  Unless you've already succumbed to finance and educated yourself about it, you probably hate it too, in the beginning.  In other words, unless someone is totally greedy, materialistic, or a "money whore" they usually try some or all of those paths above before succumbing to finance, so by the time they're ready to consider finance they're probably pretty cynical and jaded about their other options.  But then surprise!  Software Engineers in finance have the best compensation of any career track they can take (aside from winning the startup lottery) and on top of that get to work on the most interesting problems *by far*.  For example, I took graduate level algorithm design at MIT from some of the pioneers in the field, and I used that background zero times in my startup attempts, once in two years at a corporate search engine company, and about *weekly* in my finance job.  It's great.  There's so many interesting challenges and puzzles, all of which play exactly to an MIT nerd's strengths.  Imagine trying to solve the world's hardest puzzle; with math skills, ingenuity, and code as your weapons; and making millions if you succeed.

I've tried all these paths, and algorithmic trading is the best career path for me, at least now while I need to accumulate wealth in order to afford to buy my future freedom.

My own path following school was:
- startup
- corporate
- drop out
- startup
- freelance
- drop out
- finance

And my future path will probably be something like:
- drop out (hike, sail, travel, do absolutely nothing for a while)
- startup (khan academy style!)
- ?

Anyway, I've considered my options fairly exhaustively, and algorithmic trading is definitely the best career path for me right now.  And the PP is the best place for my personal investments.

[stone]

Akratic, great post- cheers for all of your insight!

It sounds as though the trading work you do is both well compensated and fullfilling. I'm suprised though that you say that freelance work is badly paid. I used to know someone who retired at 30 after doing freelance software engineering. He said that he used to come in when a huge project was stuck and unstick it and basically charge them whatever he wanted.

Your example of the bond yield curve is clearly a very benign side to trading. I'm happy to believe that that is what most of it consists of but I still have anxieties about claims that vast numbers of cancled bids are used to distort markets and inefficiencies get created inorder to be exploited. Is all of that entirely baseless?

[akratic]

...also feed into amplifying it... fairly dodgy... gaining from other people and at the same time also causing the market to be more mispriced... Is it not true that a lot of trading activity is just such volatility amplifying momentum trading?... quote stuffing...trade ahead of the tape or whatever is pure theft though isn't it?... front running funds as they try and track an index... parasitic.... anxieties about claims that vast numbers of cancled bids are used to distort markets and inefficiencies get created inorder to be exploited. Is all of that entirely baseless?

I certainly can't speak for all algo traders, and I hear the same rumors you do.  But I do know a decent number of algo traders across a few firms (citadel, goldman, prop firms, etc), and I don't know anyone making money in the ways you're describing.

In general I don't know why you'd invest all this effort into a strategy that is clearly shady and would get shut down and fined as soon as it was discovered.  Especially when there's so much money to be made through good old fashioned hard work.

If anything I think you could argue that some parts of this finance mess are over-regulated rather than under-regulated.  For example, I'm a lowly software engineer for a finance firm, and suddenly I have to deal with pages and pages of paperwork just to sign up for a Vanguard account to buy VTI with?  Does that make any sense to you?  That my personal investments would be scrutinized by all these people because my day job involves coding trading algorithms that operate on completely different time scales than I can with my personal investments?

I've even heard of people being fined and having trades reversed for what I think an educated objective third party would easily agree should have been fair game.  Stuff like buying because you thought a product was cheap, only to get the trade reversed because you got it too cheap.

By the way they regulate canceling of orders.  It is not allowed to submit an order that you do not have the intenion of trading at the time of submitting it.  And they also regulate that a certain percentage of your orders have to be filled.  And you can't trade with yourself (although all that would really accomplish is hemorrhaging fees).  And 100% of the time that an algo of mine cancels an order it's simply because it doesn't want that price anymore.  The market moves fast, remember.

It could look like I was canceling an order right before something crazy went down... which is probably exactly what happened!  My algo detected the craziness, decided it was a variety of crazy that it doesn't yet understand, and canceled the order to get out of the way.  I certainly didn't cause it.

As for amplifying volatility and parasitism like everyone thinks most algo traders (especially HFT) do... it's precisely the opposite.  If you can absorb volatility and establish a symbiotic relationship with the market, then it will pay you handsomely for the valuable risk reduction service you are providing.  

I think one reason people rarely bring this stuff up is they really have nothing to gain except more educated competition by letting people in on how things actually work.

PS: the one thing you said that I think should be fair game is I don't think funds should have a protected look at market inefficiencies/mispricings... whoever can clean it up first should get it.  I think if you forced the rest of the market to wait while the funds got to do whatever it wanted first... you'd have quite a lot of unintended consequences...

I'm suprised though that you say that freelance work is badly paid. I used to know someone who retired at 30 after doing freelance software engineering. He said that he used to come in when a huge project was stuck and unstick it and basically charge them whatever he wanted.

Perhaps if you were good at selling and good at negotiating and good at networking that could work!  I've considered giving freelancing another try with my new more mercenary mindset.  Perhaps eventually I will.  A navy seal style freelancer like you're describing could probably kill it freelancing in finance in particular.

Imagine a nerd in jeans, a baseball hat, and a hoodie trying to pull it off though.  That's precisely what I'm wearing.

[Storm]

Akratic, thanks for posting such a detailed post.  I can somewhat relate to your experience.  I consider myself a fairly top level systems engineer/architect, yet corporate America doesn't value engineers.  We are considered geeks or nerds.  I know first hand there is a huge difference in performance between engineers.  It is the same in a lot of professions, but I can accomplish in 1 week what some colleagues on my team might take 2-3 months to do.  It's all about automation and working smarter instead of harder.

When I moved to the east coast and got my first job in finance (working at UBS in Stamford) I realized that engineers were compensated fairly well in finance.  The base salary was good, and the financial incentives and bonuses were better than any other industry I've worked in.  I've worked at a few places in the finance industry from investment banks, prop trading firms, to brokerages, and I felt that finance is one of the only industries that really understands and rewards the difference between average engineers and excellent engineers.

It's not surprising that a guy with your academic background found a good niche there.  I'm also amazed that you're smart enough to put your personal money in the PP.  I know when I worked at a prop trading firm, part of their bonus structure was invested into the firm, and people boasted 20-30% annual returns through much of the early 2000s.  They had a 10 year vesting schedule and it was literally "golden handcuffs" keeping some people there because even an administrative assistant had over $1 million in this extra bonus fund.  I didn't work there after 2007 so I have no idea how they fared in 2008.

One thing I will say though is that it seems like a lot of the prop trading firms were founded by computer science geeks and not MBAs.  This probably gives them a greater appreciation for the skills you bring to the table.

In fact, I found the following story pretty inspirational:  http://www.iinews.com/site/pdfs/IIMag_I ... v_2005.pdf

This is about Thomas Petterfy who founded Timber Hill (probably the largest market maker) and Interactive Brokers.

Personally, I took the drop out approach.  I liked working in finance, but the long hours and pressure were a little bit too much for me.  I took a well paying job in healthcare, one of those corporate jobs that allows you to do a few hours of real work a day, and leave at 5, every day.  Personally it works fine for me.  My base salary is the same that I made in finance (maybe a little better) but my bonus is pretty much capped at 20%.  Not a bad trade for the amount of work I need to do.

[akratic]

inefficiencies get created inorder to be exploited...

One last thought: I can't even imagine how expensive it would be to create that big of an inefficiency in the modern market.  And how ridiculously awesome you would have to be at profiting from inefficiency to justify trying to create it.

If you're that good at profiting from inefficiency, it'd be a lot easier to just sit back and rake in the money from all the inefficiencies that everyone else who lacks omniscience creates.

[akratic]

Personally, I took the drop out approach.  I liked working in finance, but the long hours and pressure were a little bit too much for me.  I took a well paying job in healthcare, one of those corporate jobs that allows you to do a few hours of real work a day, and leave at 5, every day.

Yeah, that's exactly what I plan to do if I fail to make it big in a finance or in a startup, and I want to have a family. 

I'm 28 right now and my girlfriend is 24, so I have time.  But if I didn't have time, if she were to get pregnant tomorrow say, I would do the corporate / drop out thing and focus on my family instead of my career.

Getting to put so much effort into a place where effort can actually be rewarded is a luxury that I'm thankful for.  I had to go through a lot of bullshit and get some pretty lucky breaks to end up here.  Time to capitalize on the opportunity!

[Storm]

Personally, I took the drop out approach.  I liked working in finance, but the long hours and pressure were a little bit too much for me.  I took a well paying job in healthcare, one of those corporate jobs that allows you to do a few hours of real work a day, and leave at 5, every day.

Yeah, that's exactly what I plan to do if I fail to make it big in a finance or in a startup, and I want to have a family. 

I'm 28 right now and my girlfriend is 24, so I have time.  But if I didn't have time, if she were to get pregnant tomorrow say, I would do the corporate / drop out thing and focus on my family instead of my career.

Getting to put so much effort into a place where effort can actually be rewarded is a luxury that I'm thankful for.  I had to go through a lot of bullshit and get some pretty lucky breaks to end up here.  Time to capitalize on the opportunity!

Oblig. Breaking Bad quote: "Make hay while the sun is shining!"

[MediumTex]

akratic,

Really outstanding post.  Thanks for sharing.

The fact that you see the PP as the best fit for you and your personal money really speaks volumes.

[MachineGhost]

I think I know where you are coming from philosophically as I've been there before.  Either you can be resolutely stubborn about participating in the dysfunctional nature of our State Capitalism, or join the party.  But I do want to point out that in the old days, before central banking and before fiat currencies, there was still speculation.  It is built into the system based on weaknesses of human behavior until we all grow up and adopt something like Riegel's Valun so there are no leveraged excesses and thus, no need for speculation (although I wonder how future risks will be hedged if there are no speculators willing to take the other side of the trade...).  The most extreme "left wing" political elements would love to see futures contracts, swaps, etc. banned and eliminated because they believe speculation is damaging.  That is putting the cart before the horse of course, but they do have an unconsciousble sense that something is horribly wrong with the entire monetary/credit system.

If you haven't, I heartily recommend reading "Against The Gods: The Remarkable Story of Risk".  It will put a lot of things into perspective about all of the benefits to civilization from risk-taking and speculation.

MG

akratic, it is good to hear that my anxieties about this seem unfounded. I'm totally ignorant about all of this and it is great to hear from you guys what it is actually about. I agree with machine ghost that the PP is a crude "trading  strategy". From what I can see as a "trading strategy", the PP does benefit from the mistakes of others and much of the gains we get as PP holders come from the poor souls who buy high and sell low. But the effect on the market of people following the PP, is to smooth out such ineptitude induced volatility. By contrast, the "decision moose" and even the "leveraged etf PP" strategies not only harvest money from such volatility, they also feed into amplifying it. For those strategies, the gains do look to me to be gained by something fairly dodgy. You are gaining from other people and at the same time also causing the market to be more mispriced. Is it not true that a lot of trading activity is just such volatility amplifying momentum trading?

HFT activity such as quote stuffing so as to trade ahead of the tape or whatever is pure theft though isn't it?
Even front running funds as they try and track an index seems to me to be parasitic. I suppose the main thing is that the market does not get structured in such a way to try and maximise the ability of such parasitism. To me regulatory capture by parasites is the real danger.

As I said, I'm glad of being put right about any of this.

[stone]

Thanks so much for all this insight into a world that I know nothing about but I have heard and believed so much that has troubled me.
As an aside, the  freelance software engineer I was talking about was scruffy and an unashamed nerd. I guess the people who hired him were stuck and so he called the shots despite all the scruffy nerdiness.

I still have some unresolved issues about trading. You say that all the traders you know act on the side of rebalancing away volatility (like the 4x25% PP with tight rebalancing bands). I thought that the market also contained a large element of momentum trading too. I also thought that studies had shown that to some extent momentum traders do out perform "hedgers" who balance away volatility. At the crude retail level, the "decision moose" strategy does outperform the PP does it not?

I also don't understand why it would be so hard to create inefficiencies to exploit them. Don't firms like Goldman Sachs and JPMorgan have trillions of dollars to work with? That dwarfs the size of the markets for many commodities or individual stocks does it not? In principle isn't it true that a loss made by overselling or overbuying something could be more than recompensed by an option trade made simultaneously? I also wonder whether the algotrading inadvertantly bungs things up, that creates ripples that then get harvested by the algotraders. In good faith you inadvertantly create a hiatus like the May2010 flash crash and then in good faith harvest the wreckage?

At a more elemental level, I question what it is all in aid of. The scale of the human enterprise involved is like many Apollo missions and yet to a naïve observer it all looks avoidable. A large proportion of the most highly talented and hardworking people (such as you) are occupied with this. Imagine if the markets were specifically structured to be cheap and clunky to run. Let’s face it, the stock market in 1550 worked and had no rocket scientists involved. A hypothetical no-frills market could have crude not-for-profit computer market making and all trading would have to be through that market maker. Each trade would have a large spread (say 2%) and the profit from the spread would be transferred to the existing holders of the security as dividends. Everything could also be slowed down to a timescale of minutes rather than microsceonds. To my mind such a market would still provide all the real functions that the market is supposed to. All the rocket scientists would be freed up to do other things and all the expense would be spared .

[Lone Wolf]

3) I'm going to have to get personal to talk about why algorithmic trading is the best career for me, but this is a forum, so I suppose that is appropriate.

Great post, akratic!  Super enlightening stuff.

I'd have never thought that the PP would appeal to an algorithmic trader but after reading your reasoning I see exactly how your experiences brought you here.

I'm also rather curious about the nuts and bolts of the tools you work with.  How complex do these systems get?  Are you sitting down and working with a much larger set of libraries in a big, complex system or do you just sort of plop down in front of a blinking cursor on a brand new Perl script every couple of weeks?  (Assuming, of course, that you do not use another fine language such as Brainf*ck, of course.)