Monday 7 July 2008

The forecasting delusion

I've posted before on the failure of economic forecasters to do what they purport to do: forecasting the future (see here and here, for recent examples.)  When it comes to most economic predictions about the future, the evidence shows that it's a matter of the blind leading the sightless. 

Just look at the recent evidence gathered by Whangarei commentator Rodney Dickens [What are the economic forecasters up to now? - pdf], which compares predictions-against-performance for the top ten economists surveyed each quarter by the NZ Institute of Economic Research. The average predictions bythe economic forecasters are called the 'consensus forecasts.'

Here's what that 'consensus' predicted would happen with the unemployment rate:
                                     Unemployment-NZIER

As Dickens says, "not much use to firms and individuals who make the mistake of trying to use the consensus forecasts for business or investment risk management purposes."  Here's that same 'consensus' trying to predict consumer spending, measured against what actually happened:
                                     ConsumerSpending-NZIER

Not a complete dead loss, perhaps, but if you were relying on the "top ten" to help you out in June '02 and June '03 you'd have been in trouble.  Frankly, one can see why some contrarians make money by betting against the forecasters ... except that just occasionally the forecasters are right.
SometimesTheyAreRight-ButWhen

Despite their lack of forecasting success, economists nonetheless persist in making public predictions that are lapped up by media and business.  And many people will have locked themselves into fixed-interest mortgages on the basis of predictions of how interest rates will move, which can be contrasted to what actually happened:
                                             Interest-NZIER

The phrase "friggin' hopeless" might occur to you about now.  If it did, then think of something stronger because here's what that same "top ten consensus" made of exchange rates, as measured by the Trade Weighted Index (i.e. the weighted value of the NZD against the USD, AUD, Euro, Yen and GBP):
                                               ExchangeRate=NZIER

Not much success there either, huh?

Now, you might object about now that using the 'consensus' prediction here hides the few forecasters who've been successful.  Just to test that particular hypothesis, the chart below shows the range of predictions made by those top ten...
      

Looks lik e that hypothesis can be abandoned too. 

Now, these charts are hardly a compelling argument for the efficacy of economic forecasting.  Of course, Rodney is himself a forecaster --  his newsletter 'What really drives the NZD/USD,' he boasts, "shows that we have a much better understanding of what drives the major cycles in the NZD/USD than all the economists combined"  -- but he's at least aware that the public, that is, you, need to be aware not to take forecasters' predictions as gospel.

Unlike politicians and central planners, who do take these forecasts as gospel (and Alan Bollard, who thinks he writes the gospel) entrepreneurs themselves rely largely on their own independent judgement of what the future holds-- and it's them after all who actually move the economy and drive production.  Entrepreneurs will certainly listen to forecasters, and they definitely don't mind forecasts being taken seriously by the easily led, since it sets up opportunities to take advantage of their poor estimates.  ( What entrepreneurs are often looking for is, as Israel Kirzner explains, "unexploited opportunities for reallocating resources from [low-valued] use to another of higher value [which] offers the opportunity of pure entrepreneurial gain.  A misallocation of resources occurs because, so far, market participants have not noticed the price discrepancy involved.  This price discrepancy presents itself as an opportunity to be exploited by its discoverer.  The most impressive aspect of the market system is the tendency for such opportunities to be discovered.")

Entrepreneurs generally recognise the truth stated by Ludwig von Mises, "that the main task of action is to provide for the events of an uncertain future."  If, for example, the date of booms and busts and the like could be predicted "with apodictic certainty" according to some formula or other, then everyone would act at the same time to make it so.

    In fact, reasonable businessmen are fully aware of the uncertainty of the future. They realize that the economists do not dispense any reliable information about things to come and that all they provide is interpretation of statistical data from the past...   
    If it were possible to calculate the future state of the market, the future would not be uncertain.  There would be neither entrepreneurial loss nor profit.  What people expect from the economists is beyond the power of any mortal man.

The greatest danger of 'the forecasting delusion' is the illusion that "the future is predictable, that some formula could be substituted for the specific understanding which is the essence of entrepreneurial activity, and that familiarity with these formulas could make it possible for [bureaucratic management] to take over the conduct of business."

    The fact that the term 'speculator' is today used only with an opprobrious connotation clearly shows that our contemporaries do not even suspect in what the fundamental problem of action consists.
    Entrepreneurial judgment cannot be bought on the market. The entrepreneurial idea that carries on and brings profit is precisely that idea which did not occur to the majority. It is not correct foresight as such that yields profits, but foresight better than that of the rest.

NB: I can recommend the regular ravings of Rodney Dickens to you, from whom most of the charts above are sourced.  Click here to go straight to his latest Raving. Or click here to enter the 'Literacy Centre,' where you can access all Rodney's past Ravings.

14 comments:

Anonymous said...

PC said...
I've posted before on the failure of economic forecasters to do what they purport to do: forecasting the future (see here and here, for recent examples.)

Yes PC, failure out of how many times? Is it one out of 1000, 100 or what? You know that failure rate is relative, don't you? So, give us the figures and then we'll see?

BTW, here is a list of publications that perhaps you might want to go thru and see what evidence-based framework is all about, after all, you're the guy (not us readers) who bleats all the time here to other debators at Not PC about, but the evidence shows this, the evidence doesn't support your view, etc,...

So, if you're proponent of evidence-based, I suggest you start with the title: Financial Time Series Prediction Using Least Squares Support Vector Machines Within the Evidence Framework.

Support Vector Machines: Financial Applications

SVM is one one of the modern techniques used in forecasting. You have to understand what forecasting is first, before you rubbish them.

Peter Cresswell said...

Did you see the charts, FF?

Do you see the historic performance of your forecasters?

Do you understand what that means for the claims of the crystal-ball gazers to be able to predict the future?

You'll find the same results if you look at any series of predictions in any market, for the reasons that Mises explains above.

Frankly, I need to examine 'Financial Time Series Prediction Using Least Squares Support Vector Machines Within the Evidence Framework' less than you need to understand the implications of your forecasters' poor performance.

Just look at the evidence, man.

Matt Burgess said...

An interesting question is: why do people who are fully aware of the efficient markets hypothesis agree to make predictions?

Is it because they do not believe markets are efficient?

Or is it because they are fulfilling a demand for something they know, but presumably most others do not know, is an illusion: the ability for one person to see future asset prices in a liquid market.

It'd be interesting to ask forecasters about their motivations.

Eric Crampton said...

I'd want to know what proportion of forecasters put money behind their predictions. Why should I take seriously somebody who doesn't take himself seriously enough to bet on his own forecasts?

One of my favourite books on markets is Asensio's "Selling Short". Asensio does detailed research into firms that he thinks might be overvalued, short sells the ones he finds really are overvalued, then publishes his research. If he's right, the market moves in the direction he's put money on; if he's wrong, he loses a lot of money. That's the kind of forecasting I like.

Anonymous said...

Crampton

That's one of my favourite books. He gets a little repetitive but his main points hit hard. He's out there trying to knock over the bullshitters. For this he gets plenty of hatred...

BTW you shouldn't be putting any of your money where those dopey forecasters say to. You'd lose the lot. Far better to be giving it to me. I'd give half it back and you'd be miles ahead!

LGM

Anonymous said...

FF

No. You are wrong on this subject. The record demonstrates that those silly bullshit forecasters are not very successful in divining the future. They are wasters of time and space. They'd be better of examining the entrails of goats for their predictions- that'd have to be an improvement on their present methods.

LGM

Anonymous said...

PC said...
Did you see the charts, FF?

Yes, but that is not rigorous evidence, isn't it? That is anecdotal evidence. When the graph is being examined and put out there to peers, then peers will critique the methods being used. Scrutinize the way data is being analysed, those sorts of things.

Just in case, there are tons of forecasting methods with varying capabilities and you can't lumped them as the same. The charts that you quoted looks like to me that the forecasters used the ARMA model. There are other methods which are more robust than ARMA. WHY? Because the long term running of ARMA dies down to the mean (average) of the time-series, ie, the reason why you see the arrows tend to point to the center regardless of the rise and fall in the curve.

Sorry PC, but your graphs are not evidence at all. Anecdotal is not evidence and you know well what I mean here. You have to do better than quoting those graphs which were done (most likely) by non-forecasters who just appeared that they somehow know forecasting.

I have no faith in those celebrity forecasters that fill the interview slots in our evening TV news. These are the idiots who give forecasting a bad names.

LGM said...
The record demonstrates that those silly bullshit forecasters are not very successful in divining the future.

As I have stated above, I wouldn't treat those forecasters seriously, they're either a single method forecasters or simply guess work & rehashing old quotes.

Rodney Dickens said...

Whatever the starting point used by most of the economic forecasters the lesson is the same. The
economists are more inclined to make stupid assumptions that ensure their predictions are often well off the mark than they are motivated to build reliable forecasting frameworks or get off their butts and find out what is going on in the real world.


Now PC you've just quoted Dickens here, but he is an advocate of forecasting (bullshit as you called them). What makes you think that Dicken's method is more reliable than the bankers? Dickens criticized analysts for not find out what is going on in the real world, but I have asked you in the last thread for forecasting about why do you need to know what has been happening at all? What is the relevance of knowing the past as Dickens has pointed out about analysts who failed to do that? Because you bloody want to give a rough estimate of the future (or in other words), you want to (unknowingly) forecast the future. But you know that all economists are interested in knowing the past, aren't they?

Rodney Dickens said...
But if you are serious about business or investment risk management you are best advised to check
out the business risk management reports we produce on our website.


The guy is self-promoting his business and you take him seriously, PC? C'mon this is more like a case of one psychic criticizing another fellow psychic. I wouldn't treat Dickens seriously either.

I agree with Owen McShane that economic forecasters should publicize their methods and how they arrive in their conclusions (more like an informal peer review but not the real thing). In this way, the public can see if they're bullshitting or not.

I do agree with Dickens on one thing, that there is little research done by financial analysts themselves. Mostly rehashing old quotes to make it appear that they have done some recent research, but really there is nothing new in there. I know this because I subscribe to the NZ FundSource regular newsletter, and the most of the comments are rehashing, where it looked like that they just recycle old articles with no attempt to do research at all and Dickens is correct in his criticism.

Peter Cresswell said...

"...but that is not rigorous evidence, isn't it? That is anecdotal evidence."

You saying he faked the graphs?

Or that using "the ten or so economic forecasters surveyed by
NZIER each quarter reveals the science of economic forecasting at its best"?

Frankly, there's a bigger problem in you proving your guess at what model these "top ten" economists might have used than there is in the credibility of these charts.

Peter Cresswell said...

Sorry, that second paragraph should read, "Or that using "the ten or so economic forecasters surveyed by
NZIER each quarter [doesn't reveal] the science of economic forecasting at its best"?

Anonymous said...

FF

No again. You're still incorrect on this entire matter.

The point is that none of these guys understand economics. They say they are "economists" but they labour under all sorts of erroneous assumption and theology- mostly incorrect. In other words they are corrupted by wrong ideas.

In such cases it does not matter how clever are the mathematics they employ or how many numbers they cast into the bubbling cauldron. It does not matter what algebraic formulae they cast or what calculus they spin. It does not matter how elegant the differential equations or how brilliant the code they run. It does not matter what bright and complex formulae they recite. They are in the hopeless position of not understanding that which they make pronouncements upon.

See, it isn't a matter of dreaming up yet some other pattern of numbers and formulae. It's a matter of UNDERSTANDING economics. That means UNDERSTANDING the social science of human action.

Oh dear, that means you will need to learn economics from Reisman and Von Mises after all! Of course, I assume you would indeed like to understand economics...

LGM

Anonymous said...

Ok LGM,

How would you or PC can summarise reports for the market (suppose that you're doing it for the Ayn Rand Institute share investors) by using Mises & Reisman philosophies.

Just put a few lines below (perhaps 2 or 3) about the economic outlook (Oh, sorry I can't use that term, right?), or situation at the moment, avoiding the language use by forecasters because you think they're bullshitting.

Are you up for the challenge? It isn't difficult, I believe, so go ahead and we will see. I bet that you're/PC's misunderstand is purely semantics and I can guarantee you that whatever few sentences that you put together for your economic summary, it would be unavoidable to mention the future, whether directly or indirectly.

I await your summary, C'mon 3 lines, not that difficult.

Anonymous said...

FF

For a start I wouldn't be attempting to collectivise the entire market in three sentances. Neither should you.

What I would be doing is studying those specific companies I am interested in doing business with or investing in. That means finding out a lot of detail about them such as who the management team are, what their previous track record was like, what precursors the company requires for its activities (and what the state of their supply arrangements were like), what processes and systems are employed in the company, what sort of staff are employed (skills, industrial relations, turn-over, training, alternative employer competition for the same skills, morale, labour issues, etc.), who their customers are, what their interal and external costs are, what legislative/regulatory burdens they labour under, who the competition are, what alternatives to using that company's products and services exist, what strategy the company is following and so on. In other words I would be seeking much information about the state of that company (or those companies) which I am interested in, with as much detail as it is possible to acquire. From that point I am in a position to decide what to do about it.

What I am recommending to you is to put aside the magic numbers and actually get involved in the fray. Seek REAL information. Don't theorise in a vacuum. The way forward is to go and and find out what is going on from people who are actually in the position of holding the real knowledge and information you require. The best thing to do is learn by getting directly involved. Learning by DOING it is called.

A man who bets his money on the stocks in the absence of detailed and direct knowledge of those stocks and what they represent is little more than a blind gambler. He may as well take a punt on the horses. Same deal goes if you are trying to discuss what is happening in the economy in the absence of an UNDERSTANDING of economics.

So, what of the present situation? Von Mises and Reisman would tell you that what is presently being experienced is the result of inflation of the money supply. The credit boom is having the effect exactly as described by them. Why not look it up? Try reading about the Austrian School and its explanation of the Boom/Bust Business Cycle.

LGM

Anonymous said...

LGM said...
That means finding out a lot of detail about them such as who the management team are, what their previous track record was like, what precursors the company requires for its activities (and what the state of their supply arrangements were like), what processes and systems are employed in the company, what sort of staff are employed (skills, industrial relations, turn-over, training, alternative employer competition for the same skills, morale, labour issues, etc.), who their customers are, what their interal and external costs are, what legislative/regulatory burdens they labour under, who the competition are...

In short, what are these info useful for? Future use? If they're not for future use, then why bother collecting that info? They're simply useless if they're not for future use?

Shall we use the proper language shan't we? The whole purpose of finding out that knowledge is (pick your terms):

- rough guess of the future
- possible estimation of a future state
- what if scenario & worst case analysis of the future
- prediction of what might be likely to happen in the future
- quick forecasting of the future
- and so forth.

After all, both you and PC dismissed that knowing the volatility of the past is not a good indication of the future on this very thread we debated in May:

Risking little

What makes you think that your knowledge of :

#a) who the management team are,
#b) what their previous track record was like,
#c) what precursors the company requires for its activities
#d) turn-over
#e) competitors
...
...

are any guarantees for the future?

PC (which you agreed with) stated clearly in the Risking Little's post that , why would you assume that past volatility alone is necessarily an accurate guide to future performance? .

I will rephrase PC's question in the context of your last post:

...why would you assume that past info in steps #a), #b) and #c) are necessarily accurate guide to future performance? .

Isn't everything (analysis or fact-finding) is geared towards the future? I mean to a time-point that it is not now or today? Anything that is geared / intended towards the future is called; well pick a term from the list at the beginning of my message. If the info gathered is not geared to be used at a time-point from now on including future times, then why bother finding out those info in the first place, then? In short, if the future is undetermined (which it is), don't you think that you're wasting your time in doing such fact-finding?

Finally, here is a quote from the NZ Herald Business section (page B2) of today:

Russell senior manager of investment consulting Alister Van der Maas says, that with extremely high interest rates in NZ it means cash is extremely attractive short term. It's pretty damn good for most retail investors and quite hard for a fund manager to beat that.

I am sure that you can do a summary like the one above. I didn't mean that you write an essay, just a short market tips or something similar, but by using the Mises & Reishman's economic/market framework. I want to see if they would be any different?

So, what da you reckon? Is Alister Van der Maas's analysis useful or simply useless, ie, saying things that market participants don't need to be advised on since they already know about it from other sources?

Matt Nolan said...

Huh,

The purpose of economic forecasts is to give people a sense of where economic activity would be heading in the absense of shocks to economic activity - however shocks occur constantly so of course the numbers are never going to match.

Businesses pay for economic forecasts, not for the specific numbers, but for our description of recent data and our explanation of the risks associated with the economic outlook in the short-term. That is why whenever I forecast I spend as much time discussing risks as I do discussing what may potentially happen.

It is a bit silly to criticise economists for being wrong ex-post, the question should be, are economists using the best methods to figure out what may occur "ex-ante".

As you say, there is piles of uncertainty out there - that is why , ex-post, the forecast is always wrong. That does not mean that our description of the economic situation is not useful.

Eric said "I'd want to know what proportion of forecasters put money behind their predictions"

Many forecasting teams are part of investment teams (at least in NZ) - so the businesses return does depend strongly on the relative success of the forecasts.