Statistical analysis of phenomena that smell like chaos

Posted by Francesco Gadaleta on December 14, 2012

Is the market really predictable? How do stock prices increase? What is their dynamics?

How many have asked such questions already? How many answered. How many do believe in those answers? Here is what I think about the magics and the reality of predictions applied to markets and the stock exchange.

I am not saying anything terribly new when I speak about predictions as a cool task but also an extremely difficult one. Scientists want to predict everything, from the weather to the outcome of political elections, from disease prognosis to economy. There is a very large list of predictions that are revealed to be poor if not completely wrong. Some of them are even ridiculous. But scientists keep predicting things. Or at least they keep trying to.

Good thing is that a lot of math is usually involved even when the problem seems to be solvable with simpler models. What makes a prediction method good is often the set of assumptions that one starts with. Keeping these assumptions as close as possible to what really happens in the physical world usually leads to a more complicated model. 

The other extreme is relaxing excessively these assumptions, something that can lead to oversimplifying the model and therefore limiting its power or misinterpreting the results.

I recently attended a speech by JP Bouchaud about “The (unfortunate) complexity of economic systems” which, aroused my interest inspired me in the topic.

Despite the applicability of methods borrowed from physics, the attempt to explain the complexity of our economy with science is challenging. It is even more so, when the assumptions nobody starts with are about the consistent human factor in the (stochastic?) process of price change. The questions that Bouchaud and many others are trying to answer are “how do stock prices increase?”, “what is their dynamics?”

Answering such questions can be as fundamental as becoming super rich (for a market speculator), eventually mitigating economical crashes or, say, keeping the quality of life at a decent level for as many people as possible, for those who still care about ethics.

What Bouchaud confirmed in his speech is that the

erratic dynamics of markets are mostly endogenous and not exogenous, as one might expect.

This literally translates to the fact that no big news should be needed to change stock prices and determine huge profits. A sign of the complexity of the system is given by the very high sensitivity to small changes.

This in particular reminds me of catastrophic systems, in which a small change of some parameter leads to consistent changes within the system (due to the transition from one equilibrium to another).

An interesting observation is that while exogenous driving forces are stable, regular and steady, the resulting system dynamics is complex and intermittent. Intermittent phenomena are another sign of what mathematicians like Prof. Strogatz would define as Chaos.

Another observation of Mr Bouchaud that made me curious is regarding the collective nature of decisions taken by traders

if each trader is also influenced by what the rest of the community is doing, the overall system  will jump from optimistic - buy - to pessimistic - sell - behaviour (or the other way) even in the case of regular exogenous factors.

An explanation of the current crisis, leads back to the years before 2007, when banks were making debt on debt. According to the efficient market theory this system should have corrected itself. But as we all know, that was not the case. 

One possible reason for such a phenomenon could be that collective euphoria concealed the negative aspects of what was going on and brought the system to a state in which a little tiny, even irrelevant, news would have given rise to a global crash. And it did. Rather than statistics, I see the footprints of catastrophe theory again.

Another study that confirms, via a numerical model, Bouchaud’s views, is one titled “Unstable price dynamics as a result of information absorption in speculative markets”.

The authors of the work state that when the system is close to the point of perfect balance or critical point - that is when prices are converging to a stable value - noise can lead to instability. Susceptibility to noise increases dramatically near the critical point. This usually occurs when no news, capable of driving price oscillations, is left to be exploited. Basically, if the price is too low, traders will increase buy orders, until the price of the stock begins to rise.

As long as traders are searching for patterns in the price dynamics, they assume that the market will respond to available information. If there is no such information left apart from noise, as is the case near equilibria, traders will search for patterns into noise. This will lead them to react to random fluctuations and take decisions that might cause significant price variations. With the aforementioned numerical model, the occurrence of such a behaviour becomes quite evident: as the market price approaches an equilibrium, once all predictable information has been exploited by speculators, a little perturbation of the price (or noise) can lead to an unforeseen price change. Since the market is not well adapted to this new state, extremely large price changes can appear very frequently.

…when prices are converging to a stable value - noise can lead to instability. As long as traders are searching for patterns in the price dynamics, they assume that the market will respond to available information.

Another key observation about the stylised agent-based model, is that large returns are caused by endogenous information states that appeared less recently. Such phenomenon could be interpreted by the presence of unexpected news. Despite the time dependency of endogenous information, it might be interesting to explain the complexity of stock price fluctuations with chaos theory.

For instance, what if this sensitivity to noise were just a strange attractor? Surely, price fluctuations would still appear random, but within an attractor.

In light of these insights, approaching the problem of price or trend prediction with a merely statistical approach can limit place limitations on the reliability of the results.

Provided that the market can be predicted - and there are already serious doubts about that

  • mathematical statistics might not be sufficiently powerful for this purpose.


Before you go

If you enjoyed this post, you will love the newsletter of Data Science at Home. It’s my FREE digest of the best content in Artificial Intelligence, data science, predictive analytics and computer science. Subscribe!