Permutation and Randomization Tests for Trading System Development: Algorithms in C++

Full Disclosure: First; I have known the author for 25 years and we have collaborated on various projects including a book we co-authored.(Statistically Sound Machine Learning for Algorithmic Trading of Financial Instruments), the development of machine learning software TSSB and he was a crucial adviser on my book Evidence Based Technical Analysis. He is also a friend. Given our past relationship, if I had nothing good to say, I would say nothing. But that is far from the case here. Second, I will receive NO compensation from the sale of the book.Tim Masters, holds a Phd in Statistics and has published numerous books on subjects including artificial intelligence ( neural networks, deep learning), Statistical Prediction & Classification and trading system development and testing. This book is a significant contribution to his existing body of work and to the field of trading system development.Amazon permits you to view the contents of the book so I won't take up space doing that.The book addresses a key problem that all developers of trading systems face with a solution that permits the preservation and best possible use of the developer’s most valuable resource – the out-of-sample data. Trading system development, in a scientific manner, faces a problem not faced by many other sciences ; experimentation can not be used to develop new sets of data. We have a single body of history for empirical investigation. We have a single sample of out-of-sample data and it can only be used ONE TIME, if it is to provide a reliable and unbiased estimate of trading system performance. This all important principle, the single-usage of out-of-sample data, is unappreciated or ignored by many developers. As a result trading results are disappointing. Of course disappointment can also be the result of changed market dynamics, but there is no solution to this risk.The methods provided in this book allow developers to delay the usage of the out-of-sample data to the latest possible stage of trading system development. Because this crucial resource can only be used a single time, to obtain an unbiased estimate of a trading system true performance potential, using it too early in the development process or worse using it more than once denies the system developer key information or worse optimistically biased information.The principle statistical method used throughout the book is Monte Carlo Permutation, that involves randomizing data to destroy any authentic predictive patterns that might exist in the original (non-permuted) data. Why might someone wish to destroy the very thing we are looking for, you may ask. This is done to produce data sets that allow us to see how well a worthless , but lucky trading system, indicator or trading system development method might do. This is done many times, perhaps hundreds or thousands to develop a distribution of performance by worthless trading methods. Then by comparing how well an indicator, developed trading system or trading system development method does on actual market data we can see if the performance is better than the vast majority of approaches that are by design (data permutation) worthless. If our trading system beats 99% of what can be expected by good luck, we can conclude that our indicator, trading system or trading system development method is actually worthwhile and likely to produce profits in the future.One usage of the Monte Carlo Permutation test allows us to discover the degree of data overfitting, without using the out-of-sample data. This one use, of the nine presented is worth the price of the book. Bottom line the book would be a bargain at ten times the cost. Thus my principle critique of the book: the author gives away too much for too little, though I have benefited a greatly from his intellectual generosity.

I will admit that I have purchased every book published by Dr. Masters since "Assessing and Improving Prediction and Classification." It doesn't matter that the programming language he utilizes in his texts is C/C++. I have always found there to be so much excellent algorithmic information that the cost has always been worth it.To be honest, I approached ordering this book with some trepidation as the page count was roughly 1/3rd of his other books for a cost that was more than half of what his other books sell for. If this is a similar concern for you, please do not let it be. The content in this book is rich. I found myself re-reading many parts multiple times in my initial read-through in order to grasp the statistical concepts which Dr. Masters shares. As always, Dr. Masters shares both the code for the algorithms he presents as well as insightful commentary from decades of his "in-the-trenches" experience. In other words, I definitely felt like I got my money's worth from this book ... and then some.I will not go into a detailed review as there is a previous review here from David Aronson which I find to be both comprehensive and accurate. Bottom line, If you wish to better understand and apply permutation testing and bootstrapping to financial data, this book is a must (and pretty much the only book on the market of which I know). However, I do want to point out an additional algorithm complementary to this book which Dr. Masters has implemented just recently and made available in his VarScreen program. This algorithm is a 'stepwise option' for indicator selection based on Optimal Profit Factor. You can read about it on Page 28 of the VarScreen manual available from his website. While the details of this algorithm are, unfortunately, not present in this book, I felt that it was worthwhile to point this out in this review. The reason I learned about this recent algorithm was through email communication with Dr. Masters when I wrote to him with a couple of non-related questions. As a result, not only do purchasers of this book receive an excellent text, but they also get access to a responsive author who is more than willing to help his readers.

I have read many books about ML and algorithmic trading and this is one of my favorites.The book is well written and complex statistical concepts are explained in a very accessible way, making it possible for someone with only basic math and statistics skills to understand them.Don't be fooled by the low page count, this book is packed with valuable information and covers all the important details of the algorithms.One of the permutation tests created by the author detected a hidden selection bias problem in a trading system I'm developing. None of the other robustness tests that I've implemented before were capable of detecting that problem. That alone is worth many times the price of the book.The algorithms in the book have both a C++ and a pseudo-code version. Each step of the algorithms is also well explained in writing. Although I don't understand C++, converting the pseudo-code to Python was simple.

I've traded for 20 years, and have read all the classic books about properly testing a trading system. Dr. Masters' book adds a valuable new dimension. At first the title reference to "C++" concerned me; I'm not fluent. But I didn't need that to get the concepts. This changed the way I test, optimize, and trade.